# Connect - Full Markdown Export > This file contains all Connect documentation pages in markdown format for AI agent consumption. > Generated from 399 pages on 2026-04-10T19:52:50.797Z > Component: redpanda-connect | Version: > Site: https://docs.redpanda.com ## About This Export This export includes the **latest version** () of the Connect documentation. ### AI-Friendly Documentation Formats We provide multiple formats optimized for AI consumption: - **https://docs.redpanda.com/llms.txt**: Curated overview of all Redpanda documentation - **https://docs.redpanda.com/llms-full.txt**: Complete documentation export with all components - **https://docs.redpanda.com/redpanda-connect-full.txt**: This file - Connect documentation only - **Individual markdown pages**: Each HTML page has a corresponding .md file --- # Page 1: MCP Servers for Redpanda Connect **URL**: https://docs.redpanda.com/redpanda-connect/ai-agents/mcp-server.md --- # MCP Servers for Redpanda Connect --- title: MCP Servers for Redpanda Connect latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: mcp-server/index page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: mcp-server/index.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/ai-agents/pages/mcp-server/index.adoc description: Expose your Redpanda Connect configurations and internal tools as AI-consumable HTTP endpoints. page-git-created-date: "2025-09-30" page-git-modified-date: "2025-09-30" --- - [MCP Server Overview](overview/) Discover the business value, use cases, and architecture for Redpanda Connect MCP servers. - [MCP Server Quickstart](quickstart/) Expose Redpanda Connect components as AI-consumable tools using the Model Context Protocol (MCP). - [MCP Tool Execution and Components](concepts/) Understand the MCP execution model and choose the right component type for your tools. - [Create an MCP Tool](create-tool/) Create an MCP tool with the correct YAML structure, metadata, and parameter mapping. - [MCP Tool Design](best-practices/) Design effective MCP tool interfaces with clear names, descriptions, and input properties. - [MCP Tool Patterns](tool-patterns/) Cookbook of reusable patterns for building MCP tools with Redpanda Connect. - [Troubleshoot MCP Servers](troubleshooting/) Diagnose and fix common issues when building and running MCP servers with Redpanda Connect. --- # Page 2: MCP Tool Design **URL**: https://docs.redpanda.com/redpanda-connect/ai-agents/mcp-server/best-practices.md --- # MCP Tool Design --- title: MCP Tool Design latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: mcp-server/best-practices page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: mcp-server/best-practices.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/ai-agents/pages/mcp-server/best-practices.adoc description: Design effective MCP tool interfaces with clear names, descriptions, and input properties. page-topic-type: best-practices personas: ai_agent_developer learning-objective-1: Write tool names and descriptions that help AI clients select the right tool learning-objective-2: Define input properties with appropriate types and constraints learning-objective-3: Design focused tools that complete quickly page-git-created-date: "2026-01-13" page-git-modified-date: "2026-01-13" --- After [creating your first tool](../create-tool/), apply these design guidelines so AI clients can discover, understand, and invoke your tools correctly. After reading this page, you will be able to: - Write tool names and descriptions that help AI clients select the right tool - Define input properties with appropriate types and constraints - Design focused tools that complete quickly ## [](#tool-discovery)Tool discovery AI clients use tool names and descriptions to decide which tool to call. Good metadata helps AI select the right tool and provide correct inputs. The `label` field defines the tool name. The `meta.mcp` block defines the description and input properties. ```yaml label: get-weather meta: tags: [ weather, api, example ] mcp: enabled: true description: "Get current weather for a city. Returns temperature, humidity, and conditions." properties: - name: city type: string description: "City name (e.g., 'London', 'Tokyo', 'New York')" required: true ``` For the complete field reference, see [MCP metadata fields](../create-tool/#mcp-metadata). ### [](#choose-a-clear-tool-name)Choose a clear tool name AI clients see the `label` value when selecting tools, so make it descriptive and consistent. The name should reflect the tool’s primary action and target resource. Focus on clarity over brevity. For example, use `get-weather-forecast` instead of just `get-data`. ### [](#write-effective-descriptions)Write effective descriptions Tool descriptions help AI clients decide when to use your tool. Start with an action verb and explain what the tool returns. ```yaml description: "Get current weather for a city. Returns temperature in Celsius and Fahrenheit, humidity percentage, and weather description." ``` Include any limitations or requirements in the description: ```yaml description: "Search product catalog by name or category. Returns up to 10 results. Requires at least one search term." ``` ### [](#define-input-properties-clearly)Define input properties clearly Each property needs a name, type, description, and required status. Use `string`, `number`, or `boolean` types. Include example values in descriptions: ```yaml properties: - name: city type: string description: "City name (e.g., 'London', 'New York', 'Tokyo')" required: true - name: units type: string description: "Temperature units: 'celsius' or 'fahrenheit'. Defaults to 'celsius'." required: false - name: limit type: number description: "Maximum results to return (1-100). Defaults to 10." required: false - name: include_forecast type: boolean description: "If true, include 5-day forecast. Defaults to false." required: false ``` Mark properties as required only if the tool cannot function without them: ```yaml properties: - name: city type: string description: "City name to get weather for" required: true ``` Optional properties should have sensible defaults: ```yaml - name: units type: string description: "Temperature units. Defaults to 'celsius'." required: false ``` For patterns that apply defaults and validate values at runtime, see [input validation patterns](../tool-patterns/#input-validation). ## [](#tool-execution)Tool execution ### [](#keep-tools-focused)Keep tools focused Each tool should do one thing well. If you find yourself adding multiple unrelated operations, split them into separate tools. Tools that do too much or have vague purposes cause problems because AI clients rely on descriptions to choose tools. Vague descriptions lead to wrong tool selection. Also, tools that do too much are harder to test and debug. Write descriptions that clearly state what the tool does, what input it needs, and what it returns. If a tool is doing multiple things, split it into focused tools. ### [](#design-for-quick-completion)Design for quick completion MCP tools should complete quickly. AI clients wait for responses, and long-running tools cause poor user experiences. Tools that wait indefinitely, poll continuously, or never return cause problems because MCP tools use a request/response model. A tool that never completes will time out and fail, and resources remain allocated while waiting. Follow these guidelines: - Set explicit timeouts on all external calls. For timeout options, see [`http` processor](../../../components/processors/http/). - Avoid unbounded reads. Read N messages, not all messages. - Consider pagination for large datasets. - Return partial results if full processing takes too long. For patterns that handle timeout failures gracefully, see [error handling patterns](../tool-patterns/#error-handling). ## [](#complete-example)Complete example This example combines all the best practices: ```yaml label: search-customer-orders processors: - mutation: | let customer_id = this.customer_id | "" let status = this.status | "" let limit = (this.limit | 10).number() root = { "orders": [ { "order_id": "ord_001", "customer_id": $customer_id, "status": if $status != "" { $status } else { "delivered" }, "total": 125.99 } ], "count": 1, "filters_applied": { "customer_id": $customer_id, "status": $status, "limit": $limit } } meta: tags: [ orders, database, production ] mcp: enabled: true description: "Search customer orders by customer ID, date range, or status. Returns order summaries with totals. Maximum 50 results per query." properties: - name: customer_id type: string description: "Customer ID (e.g., 'cust_12345'). Required if no other filters provided." required: false - name: status type: string description: "Order status filter: 'pending', 'shipped', 'delivered', or 'cancelled'." required: false - name: start_date type: string description: "Start date for date range filter (ISO 8601, e.g., '2024-01-01')." required: false - name: end_date type: string description: "End date for date range filter (ISO 8601, e.g., '2024-12-31')." required: false - name: limit type: number description: "Maximum results to return (1-50). Defaults to 10." required: false ``` ## [](#next-steps)Next steps - [Use secrets](../create-tool/#secrets): Store credentials securely - [Use tags](../create-tool/#tool-filtering): Organize and filter tools - [MCP Tool Patterns](../tool-patterns/): Find reusable patterns including validation, error handling, and response formatting - [Troubleshoot MCP Servers](../troubleshooting/): Diagnose common issues --- # Page 3: MCP Tool Execution and Components **URL**: https://docs.redpanda.com/redpanda-connect/ai-agents/mcp-server/concepts.md --- # MCP Tool Execution and Components --- title: MCP Tool Execution and Components latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: mcp-server/concepts page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: mcp-server/concepts.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/ai-agents/pages/mcp-server/concepts.adoc description: Understand the MCP execution model and choose the right component type for your tools. page-topic-type: concepts personas: ai_agent_developer, streaming_developer learning-objective-1: Describe the request/response execution model learning-objective-2: Explain how Redpanda Connect components map to MCP tools learning-objective-3: Choose the right component type (processor, input, output, cache) for a use case page-git-created-date: "2026-01-13" page-git-modified-date: "2026-01-13" --- This page explains how MCP tools execute and how to choose the right component type for your use case. After reading this page, you will be able to: - Describe the request/response execution model - Explain how Redpanda Connect components map to MCP tools - Choose the right component type (processor, input, output, cache) for a use case ## [](#execution-model)The MCP execution model When an AI client calls an MCP tool, the MCP server handles the request in a specific sequence. The execution follows these steps: 1. The AI client sends a JSON request to the MCP server with the tool name and parameters. 2. The MCP server finds the corresponding component configuration. 3. The MCP server executes the component with the input data. 4. The component runs to completion and returns a result. 5. The MCP server sends the result back to the AI client. 6. The component instance is torn down. This execution model has several important characteristics: - Stateless execution: Each tool invocation is independent. Tools do not maintain state between calls. If you need state, use an external store such as a [cache](../../../components/caches/about/), database, or Redpanda topic. - Synchronous by default: Tools run synchronously from the AI client’s perspective. The client waits for the response before continuing. - Timeout boundaries: Tools should complete quickly. Long-running operations should be avoided or handled asynchronously. Set explicit timeouts on external calls. - No continuous processing: Unlike a traditional Redpanda Connect pipeline, MCP tools do not poll for messages or maintain connections between invocations. They start, execute, and stop. ## [](#how-components-map-to-mcp-tools)How components map to MCP tools Each MCP tool is implemented as a single Redpanda Connect component. The component type determines what the tool can do. The following table shows which component types are available and their purposes: | Component Type | Purpose as an MCP Tool | | --- | --- | | Processor | Transforms, validates, or computes data. Calls external APIs. Returns results to the AI client. | | Output | Writes data to external systems (Redpanda topics, databases, APIs). Can include processors for transformation before writing. | | Input | Reads data from external systems. Returns the read data to the AI client. | | Cache | Stores and retrieves data for use by other tools. | Most MCP tools are processors. Use outputs when you need to write data. Use inputs when you need to read from external data sources. ## [](#component-selection)Choose the right component type Every MCP tool is implemented as a single component. Choosing the right component type is a critical design decision that affects what your tool can do and how it behaves. ### [](#decision-framework)Decision framework To choose the right component type, ask what the tool’s primary purpose is. Use the following table to match your tool’s intent to a component type: | Question | Component Type | | --- | --- | | Does the tool compute or transform data and return results? | Processor | | Does the tool call external APIs and return the response? | Processor | | Does the tool write data to an external system (database, topic, API)? | Output | | Does the tool read data from an external source and return it? | Input | | Does the tool store and retrieve temporary data for other tools? | Cache | The core principle is to choose the component type that matches the tool’s primary intent. ### [](#processor-tools)Processor tools Processor tools transform, validate, compute, or fetch data and return results to the AI client. This is the most common tool type. See the [processors reference](../../../components/processors/about/) for available processors. #### [](#when-to-choose-a-processor-tool)When to choose a processor tool Choose a processor tool when the tool’s purpose is to compute or transform data, call an external API and return the response, or validate inputs and return errors or results. #### [](#use-case-fetch-and-transform-external-data)Use case: Fetch and transform external data Consider a scenario where an AI agent needs current weather data to answer a user’s question about whether to bring an umbrella. The following prompts should trigger this type of tool: - "What’s the weather in Berlin?" - "Is it raining in Tokyo right now?" - "Get me the current temperature for Seattle." A processor is the right choice because the tool fetches data from an API, transforms it into a useful format, and returns it. #### [](#use-case-validate-and-normalize-data)Use case: Validate and normalize data Consider a scenario where an AI agent needs to validate user-submitted data and return structured feedback about any issues. The following prompts should trigger this type of tool: - "Validate this customer record before saving." - "Check if this order has all required fields." - "Normalize this JSON and tell me what’s missing." A processor is the right choice because the tool examines data, applies validation rules, and returns results. No data is written anywhere. ### [](#output-tools)Output tools Output tools write data to external systems. Use them when the primary purpose is to create a side effect such as persisting data, publishing an event, or triggering an action. See the [outputs reference](../../../components/outputs/about/) for available outputs. #### [](#when-to-choose-an-output-tool)When to choose an output tool Choose an output tool when the tool’s purpose is to write data to Redpanda, a database, or an external API. The side effect (writing) should be the primary intent, not incidental. You can use `processors:` within the output to transform data before writing. Output tools are appropriate when you want the AI to trigger real-world actions. #### [](#understanding-tool-response-vs-side-effect)Understanding tool response vs. side effect Output tools have two outcomes: the side effect (data is written to the destination) and the tool response (the AI client receives confirmation that the write succeeded). The AI client does not receive the written data back. It receives status information. If you need to return the written data, consider using a processor tool instead. #### [](#use-case-publish-events-to-redpanda)Use case: Publish events to Redpanda Consider a scenario where an AI agent needs to publish order events to Redpanda for downstream processing. The following prompts should trigger this type of tool: - "Publish this order to Redpanda." - "Send the order event to the orders topic." - "Record this new order for processing." An output is the right choice because the purpose is to write data to Redpanda. The AI needs to create a persistent record, not just compute something. #### [](#use-case-transform-and-publish)Use case: Transform and publish Output components can include a `processors:` section that transforms data before writing to the destination. This is a single output component, not a combination of component types. Consider a scenario where an AI agent asks an LLM to summarize a document, then stores both the original and summary in Redpanda. The following prompts should trigger this type of tool: - "Summarize this document and save it." - "Process this feedback with GPT and store the analysis." - "Analyze this text and publish the results." An output with processors is the right choice because the primary intent is to store data. The processors provide pre-processing before writing. The execution flow for this pattern is as follows: 1. AI client calls the tool with input data. 2. The `processors` section transforms the data. 3. The output component writes the transformed data to the destination. 4. The tool returns a response to the AI client. For implementation examples, see [outputs with processors](../tool-patterns/#outputs-with-processors) in the tool patterns guide. ### [](#input-tools)Input tools Input tools read data from external sources and return it to the AI client. They’re useful when you need to query or fetch existing data. See the [inputs reference](../../../components/inputs/about/) for available inputs. #### [](#when-to-choose-an-input-tool)When to choose an input tool Choose an input tool when the tool’s purpose is to read and return data from an external source, consume messages from a Redpanda topic, or build a query-style tool that retrieves existing data. #### [](#bounded-vs-unbounded-reads)Bounded vs. unbounded reads Input tools must return a finite result. Use bounded reads that fetch a specific number of messages or read until a condition is met. For example, "get me the latest N events" or "read messages from the last hour". Unbounded reads that poll continuously are not appropriate for MCP tools because the tool would never return a response to the AI client. #### [](#latency-and-scope-considerations)Latency and scope considerations Keep these factors in mind when building input tools: - Input tools may have variable latency depending on the data source. - Scope your reads appropriately. Don’t try to read entire topics. - Consider consumer group behavior: with a consumer group, each invocation advances through the stream. Without one, each invocation may read the same data. #### [](#use-case-query-recent-events)Use case: Query recent events Consider a scenario where an AI agent needs to retrieve recent user activity events to understand user behavior. The following prompts should trigger this type of tool: - "Show me recent user events." - "Get the last 10 login events." - "What events happened in the user-events topic recently?" An input is the right choice because the tool reads from an existing data source (topic) and returns what it finds. ### [](#cache-tools)Cache tools Cache tools store and retrieve temporary data that other tools can access. They’re useful for sharing state between tool calls or storing frequently accessed data. See the [caches reference](../../../components/caches/about/) for available caches. #### [](#when-to-choose-a-cache-tool)When to choose a cache tool Choose a cache tool when the tool’s purpose is to store temporary data that expires after a set time, share state between multiple tool calls in a conversation, or reduce repeated calls to slow external APIs by caching results. #### [](#use-case-session-state-management)Use case: Session state management Consider a scenario where an AI agent needs to remember user preferences across multiple tool calls within a conversation. The following prompts should trigger this type of tool: - "Remember that I prefer metric units." - "Store my timezone as America/New\_York." - "Save this search filter for later." A cache is the right choice because the data is temporary, session-scoped, and needs to be accessible by other tools during the conversation. #### [](#use-case-api-response-caching)Use case: API response caching Consider a scenario where an AI agent frequently looks up the same reference data (like exchange rates or product catalogs) and you want to avoid repeated API calls. The following prompts should trigger cache usage: - "Get the current exchange rate" (cached for 5 minutes) - "Look up product details" (cached for 1 hour) - "Check inventory levels" (cached briefly to reduce load) A cache is the right choice because you want to store API responses temporarily and serve them on subsequent requests without hitting the external API again. ### [](#component-selection-summary)Component selection summary The following table summarizes when to use each component type: | Component | Primary Intent | Example Tools | Returns | | --- | --- | --- | --- | | Processor | Compute, transform, validate, fetch | Weather lookup, data validation, API calls | Computed result | | Output | Write data with side effects | Publish events, store records, trigger webhooks | Write confirmation | | Output + processors | Transform then write | Summarize and store, enrich and publish | Write confirmation | | Input | Read and return data | Query recent events, search logs | Retrieved data | | Cache | Store and retrieve temporary data | Session state, API response caching | Cached value or confirmation | For implementation examples and common patterns, see [MCP Tool Patterns](../tool-patterns/). ## [](#support-and-licensing)Component support levels and licensing Redpanda Connect components have different support levels and licensing requirements. Understanding these helps you make informed decisions when building MCP tools. ### [](#support-levels)Support levels Components fall into two support categories: Certified components Officially supported by Redpanda. These components receive regular updates, bug fixes, and are tested for production use. Use certified components when reliability and support are priorities. Community components Contributed by the community. These may have varying levels of maintenance and testing. Review the component documentation and consider your risk tolerance before using in production. ### [](#enterprise-licensing)Enterprise licensing Some components require a Redpanda Enterprise license. These include certain cloud-provider integrations, advanced security features, and some enterprise data source connectors. If you use an Enterprise-licensed component without a valid license, the MCP server will not start, and you’ll see an error indicating the missing license. The [component reference documentation](../../../components/about/) displays support level and license badges on each component page. For more information about licenses, see [Enterprise Licensing](../../../get-started/licensing/). ## [](#next-steps)Next steps Continue your learning journey with these resources: - [Create an MCP Tool](../create-tool/): Create custom tools step by step - [MCP Tool Design](../best-practices/): Apply naming and design guidelines - [MCP Tool Patterns](../tool-patterns/): Find reusable patterns - [Troubleshoot MCP Servers](../troubleshooting/): Diagnose common issues --- # Page 4: Create an MCP Tool **URL**: https://docs.redpanda.com/redpanda-connect/ai-agents/mcp-server/create-tool.md --- # Create an MCP Tool --- title: Create an MCP Tool latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: mcp-server/create-tool page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: mcp-server/create-tool.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/ai-agents/pages/mcp-server/create-tool.adoc description: Create an MCP tool with the correct YAML structure, metadata, and parameter mapping. page-topic-type: how-to personas: ai_agent_developer, streaming_developer, data_engineer learning-objective-1: Create a tool file with the correct structure and MCP metadata learning-objective-2: Map MCP parameters to component configuration fields using Bloblang learning-objective-3: Test tools locally before connecting AI clients page-git-created-date: "2026-01-13" page-git-modified-date: "2026-01-13" --- After [deploying your first MCP server](../quickstart/), create custom tools that AI clients can discover and invoke. This guide walks you through the process using any Redpanda Connect component. After reading this page, you will be able to: - Create a tool file with the correct structure and MCP metadata - Map MCP parameters to component configuration fields using Bloblang - Test tools locally before connecting AI clients ## [](#prerequisites)Prerequisites - At least version 4.66.1 of [Redpanda Connect installed](../../../install/rpk/) - You can describe the MCP execution model (see [The MCP execution model](../concepts/#execution-model)) - You have chosen the right component type for your use case (see [Choose the right component type](../concepts/#component-selection)) ## [](#set-up-the-project)Set up the project If you don’t have an MCP server project yet, create one: ```bash rpk connect mcp-server init cd ``` This command creates the following structure: ```text / ├── resources/ ├── caches/ # Cache components ├── inputs/ # Input components ├── outputs/ # Output components └── processors/ # Processor components (most common) ``` The `resources/` directory holds your MCP tool files, organized by component type. ## [](#create-the-tool-file)Create the tool file Create a YAML file in the appropriate directory. For example, to create a processor tool: ```bash touch resources/processors/.yaml ``` The file name can be anything, but use descriptive names that reflect the tool’s purpose. Each YAML file must contain exactly one component. The directory in your project determines the component type: | Directory | Component type | | --- | --- | | resources/processors/ | Processor | | resources/inputs/ | Input | | resources/outputs/ | Output | | resources/caches/ | Cache | For example, a file in `resources/processors/` defines a processor component. Do not mix component types in the same file. ## [](#anatomy)Add the tool structure An MCP tool wraps a [Redpanda Connect component](../../../components/about/) and exposes it to AI clients. Each tool file has three parts: - **Label**: The tool name AI clients see - **Component configuration**: A Redpanda Connect component (processor, input, output, or cache) - **MCP metadata**: The tool’s purpose and input parameters Here’s an example using the [`sql_select` processor](../../../components/processors/sql_select/): ```yaml label: lookup-customer (1) sql_select: (2) driver: postgres dsn: "${DATABASE_URL}" table: customers columns: ["id", "name", "email", "plan"] where: id = ? args_mapping: '[this.customer_id]' meta: (3) mcp: enabled: true description: "Look up a customer by ID and return their profile." properties: - name: customer_id type: string description: "The customer's unique identifier" required: true ``` | 1 | The label becomes the tool name that AI clients see and invoke. | | --- | --- | | 2 | The sql_select processor queries a PostgreSQL database. | | 3 | The MCP metadata tells AI clients what this tool does and what parameters it accepts. | The following sections show how to structure tools for each component type. ### [](#label-naming-rules)Label naming rules The `label` field (tool name) must follow these rules: - Lowercase letters, numbers, underscores, and hyphens only (`a-z`, `0-9`, `_`, `-`) - Cannot start with an underscore - No spaces or special characters Valid examples: `get-weather`, `lookup_customer`, `send-notification-v2` ### [](#component-types)Component types [Processors](../../../components/processors/about/) transform, filter, or enrich data. Use a `processors:` array with one or more processors: ```yaml label: enrich-order processors: - http: url: "https://api.example.com/lookup" verb: GET meta: mcp: enabled: true description: "Enrich order with customer data" ``` [Inputs](../../../components/inputs/about/) read data from sources, [outputs](../../../components/outputs/about/) write data to destinations, and [caches](../../../components/caches/about/) store and retrieve data. Define these components directly at the top level. Do not wrap components in `input:`, `output:`, or `cache:` blocks. This syntax is for pipelines, not MCP tools. Input tool ```yaml label: read-events redpanda: (1) seed_brokers: ["${REDPANDA_BROKERS}"] topics: ["events"] consumer_group: "mcp-reader" meta: mcp: enabled: true description: "Read events from Redpanda" ``` | 1 | The component name (redpanda) is at the top level, not wrapped in input:. | | --- | --- | Output tool ```yaml label: publish-event redpanda: seed_brokers: ["${REDPANDA_BROKERS}"] topic: "processed-events" meta: mcp: enabled: true description: "Publish event to Redpanda" ``` Cache tool ```yaml label: session-cache memory: default_ttl: 300s meta: mcp: enabled: true description: "In-memory cache for session data" ``` Outputs can include a `processors:` section to transform data before publishing: Output with processors ```yaml label: publish-with-timestamp processors: - mutation: | root = this root.published_at = now() redpanda: seed_brokers: ["${REDPANDA_BROKERS}"] topic: "processed-events" meta: mcp: enabled: true description: "Add timestamp and publish to Redpanda" ``` For more examples, see [outputs with processors](../tool-patterns/#outputs-with-processors). ### [](#mcp-metadata)MCP metadata fields The `meta.mcp` block defines how AI clients discover and interact with your tool. These fields control tool visibility, naming, and input parameters. | Field | Required | Description | | --- | --- | --- | | enabled | Yes | Set to true to expose this component as an MCP tool. Set to false to disable without deleting the configuration. | | description | Yes | Explains what the tool does and what it returns. AI clients use this to decide when to call the tool. | | properties | No | Array of input parameters the tool accepts. See Property fields for the fields in each property. | | tags | No | Array of strings for categorizing tools. Use with --tag flag to filter which tools are exposed. | #### [](#property-fields)Property fields Each entry in the `properties` array defines an input parameter: | Field | Required | Description | | --- | --- | --- | | name | Yes | Parameter name. | | type | Yes | Data type. Must be one of: string, number, or boolean. | | description | Yes | Explains what the parameter is for. Include example values and any constraints. | | required | Yes | Set to true if the tool cannot function without this parameter. | #### [](#property-restrictions)Property restrictions by component type Different component types have different property capabilities when exposed as MCP tools: | Component Type | Property Support | Details | | --- | --- | --- | | input | Only supports the count property | AI clients can specify how many messages to read, but you cannot define custom properties. | | cache | No custom properties | Properties are hardcoded to key and value for cache operations. | | output | Custom properties supported | AI sees properties as an array for batch operations: [{prop1, prop2}, {prop1, prop2}]. | | processor | Custom properties supported | You can define any properties needed for data processing operations. | ## [](#parameter-mapping)Map parameters to component fields When an AI client calls your tool, the `arguments` object becomes the message body. You can access these arguments using [Bloblang](../../../guides/bloblang/about/), but the syntax depends on where you’re using it: - **Inside Bloblang contexts** (mutation, mapping, args\_mapping): Use `this.field_name` - **Inside string fields** (URLs, topics, headers): Use interpolation `${! json("field_name") }` ### [](#in-bloblang-contexts)In Bloblang contexts Use `this` to access message fields directly in processors like `mutation`, `mapping`, or in `args_mapping` fields: ```yaml mutation: | root.search_query = this.query.lowercase() root.max_results = this.limit.or(10) ``` ```yaml sql_select: table: orders where: customer_id = ? AND status = ? args_mapping: '[this.customer_id, this.status.or("active")]' ``` ### [](#in-string-fields-interpolation)In string fields (interpolation) Use `${! …​ }` interpolation to embed Bloblang expressions inside string values like URLs or topic names: ```yaml http: url: 'https://api.weather.com/v1/current?city=${! json("city") }&units=${! json("units").or("metric") }' ``` ```yaml redpanda: seed_brokers: ["${REDPANDA_BROKERS}"] (1) topic: '${! json("topic_name") }' (2) ``` | 1 | ${VAR} without ! is environment variable substitution, not Bloblang. | | --- | --- | | 2 | ${! …​ } with ! is Bloblang interpolation that accesses message data. | > 💡 **TIP** > > For more on Bloblang syntax, see [Bloblang](../../../guides/bloblang/about/). For interpolation details, see [Interpolation](../../../configuration/interpolation/). ### [](#provide-defaults-for-optional-parameters)Provide defaults for optional parameters Use `.or(default)` to handle missing optional parameters: ```yaml mutation: | root.city = this.city # Required - will error if missing root.units = this.units.or("metric") # Optional with default root.limit = this.limit.or(10).number() # Optional, converted to number ``` Declare which parameters are required in your `meta.mcp.properties`: ```yaml properties: - name: city type: string description: "City name to look up" required: true - name: units type: string description: "Temperature units: 'metric' or 'imperial' (default: metric)" required: false ``` ## [](#secrets)Use secrets and environment variables Never hardcode credentials, API keys, or connection strings in your tool files. Use [environment variable substitution](../../../configuration/interpolation/#environment-variables) to inject secrets at runtime. Reference environment variables using `${VARIABLE_NAME}` syntax. Redpanda Connect replaces these placeholders when loading the configuration: ```yaml http: url: "https://api.example.com/data" headers: Authorization: "Bearer ${API_TOKEN}" sql_select: driver: postgres dsn: "${DATABASE_URL}" table: customers ``` > 📝 **NOTE** > > `${VAR}` is environment variable substitution (resolved at startup). `$\{! expr }` is Bloblang interpolation (resolved at runtime from message data). See [Map parameters to component fields](#parameter-mapping) for the difference. Set environment variables before starting the MCP server: ```bash export API_TOKEN="your-secret-token" export DATABASE_URL="postgres://user:password@localhost:5432/mydb" rpk connect mcp-server --address localhost:4195 ``` For production deployments, load secrets from a secrets manager or inject them from your deployment system rather than exporting them in a shell. For naming conventions and security guidelines, see [MCP Tool Design](../best-practices/). ## [](#test-the-tool)Test the tool 1. Navigate to the root of your MCP project directory. 2. Lint your configuration: ```bash rpk connect mcp-server lint ``` 3. Start the MCP server: ```bash rpk connect mcp-server --address localhost:8080 ``` 4. Test tool calls using curl: ```bash #!/bin/bash # Start SSE connection and capture session ID exec 3< <(curl -s -N http://localhost:8080/sse) read -r line <&3 # event: endpoint read -r line <&3 # data: /sse?sessionid=XXX SESSION_ID=$(echo "$line" | sed 's/.*sessionid=//') echo "Session ID: $SESSION_ID" # Initialize the session curl -s -X POST "http://localhost:8080/message?sessionid=$SESSION_ID" \ -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"test","version":"1.0"}}}' sleep 1 # Call your tool (replace with your tool name and arguments) curl -s -X POST "http://localhost:8080/message?sessionid=$SESSION_ID" \ -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"lookup-customer","arguments":{"customer_id":"cust_12345"}}}' # Read SSE responses sleep 2 while read -r -t 1 line <&3; do echo "$line" done exec 3<&- ``` 5. Connect to an AI client and verify the tool appears. For example, with Claude Code: ```bash claude mcp add local -- npx mcp-remote http://localhost:8080/sse claude /mcp ``` 6. Test end-to-end with realistic prompts to verify the AI client uses your tool correctly. ## [](#tool-filtering)Control which tools are exposed By default, `rpk connect mcp-server` exposes all tools in your project that have `meta.mcp.enabled: true`. Use tags to selectively expose subsets of tools. Add tags to your tool’s metadata: ```yaml meta: mcp: enabled: true description: "Query production database" tags: - production - database ``` Start the server with the `--tag` flag to expose only tools matching specific tags. The flag supports regular expressions: ```bash # Expose only tools tagged "production" rpk connect mcp-server --tag production # Expose tools with tags starting with "db-" rpk connect mcp-server --tag "db-.*" ``` For guidance on organizing tools by environment, feature, or access level, see [Tag strategies](../best-practices/#tags). ## [](#complete-example)Complete example Here’s a complete tool that wraps the `http` processor to fetch weather data: ```yaml label: get-weather processors: # Validate and sanitize input - label: validate_city mutation: | root.city = if this.city.or("").trim() == "" { throw("city is required") } else { this.city.trim().lowercase().re_replace_all("[^a-z\\s\\-]", "") } root.units = this.units.or("metric") # Fetch weather data - label: fetch_weather try: - http: url: 'https://wttr.in/${! json("city") }?format=j1' verb: GET timeout: 10s - mutation: | root.weather = { "location": this.nearest_area.0.areaName.0.value, "country": this.nearest_area.0.country.0.value, "temperature_c": this.current_condition.0.temp_C, "temperature_f": this.current_condition.0.temp_F, "condition": this.current_condition.0.weatherDesc.0.value, "humidity": this.current_condition.0.humidity, "wind_kph": this.current_condition.0.windspeedKmph } # Handle errors gracefully - label: handle_errors catch: - mutation: | root.error = true root.message = "Failed to fetch weather: " + error() meta: mcp: enabled: true description: "Get current weather for a city. Returns temperature, conditions, humidity, and wind speed." properties: - name: city type: string description: "City name (e.g., 'London', 'New York', 'Tokyo')" required: true - name: units type: string description: "Temperature units: 'metric' or 'imperial' (default: metric)" required: false ``` ## [](#next-steps)Next steps - [MCP Tool Design](../best-practices/): Apply naming and design guidelines. - [MCP Tool Patterns](../tool-patterns/): Find patterns for databases, APIs, and Redpanda. - [Troubleshoot MCP Servers](../troubleshooting/): Diagnose common issues. - [Components Catalog](../../../components/about/): Browse all available components. --- # Page 5: MCP Server Overview **URL**: https://docs.redpanda.com/redpanda-connect/ai-agents/mcp-server/overview.md --- # MCP Server Overview --- title: MCP Server Overview latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: mcp-server/overview page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: mcp-server/overview.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/ai-agents/pages/mcp-server/overview.adoc description: Discover the business value, use cases, and architecture for Redpanda Connect MCP servers. page-topic-type: overview personas: ai_agent_developer, evaluator learning-objective-1: Explain what an MCP server is and how tools differ from pipelines learning-objective-2: Identify use cases where MCP servers provide business value learning-objective-3: Describe how MCP tools expose Redpanda Connect components to AI page-git-created-date: "2025-09-30" page-git-modified-date: "2026-01-13" --- This page introduces MCP servers and helps you decide if they’re right for your use case. After reading this page, you will be able to: - Explain what an MCP server is and how tools differ from pipelines - Identify use cases where MCP servers provide business value - Describe how MCP tools expose Redpanda Connect components to AI ## [](#what-is-mcp)What is MCP? MCP (Model Context Protocol) is an open standard that lets AI agents use tools. Think of it like a universal adapter: instead of building custom integrations for every AI system, you define your tools once using MCP, and any MCP-compatible AI client can discover and use them. Without MCP, connecting AI to your business systems requires custom API code, authentication handling, and response formatting for each AI platform. With MCP, you describe what a tool does and what inputs it needs, and the protocol handles the rest. ## [](#what-is-redpanda-connect-mcp-server)What is Redpanda Connect MCP server? Redpanda Connect’s MCP server lets you expose data pipelines and automations as AI-consumable tools (MCP tools) with no custom API code. You write tools as YAML configurations, and the MCP server makes them available to AI agents. This means AI agents can: - Pull information from databases, APIs, message queues, or Redpanda topics - Publish messages, update records, trigger workflows, or call external services - Only access the specific tools you configure and enable If you already use Redpanda Connect for data integration and processing, adding an MCP server gives AI agents controlled access to your existing systems without extra development effort. ## [](#mcp-tools-are-not-pipelines)MCP tools are not pipelines If you already use Redpanda Connect, you might wonder how MCP tools differ from pipelines. A pipeline is a continuous data flow: data streams from an input, through processors, to an output. The pipeline runs indefinitely, processing many messages over time. An MCP tool is different. It’s a single component that executes on demand when called by an AI client. The tool starts, runs, and completes for each invocation. There is no persistent state between calls. Think of it like calling a function rather than running a service. This request/response pattern is what makes MCP tools useful for AI agents: the agent asks a question, the tool runs, and it returns an answer. ## [](#use-cases)Use cases | Category | Example prompts | | --- | --- | | Operational monitoring | Check partition lag for customer-events topicShow me the top 10 producers by message volume todayGet schema registry health status | | Data enrichment and analysis | Fetch user profile data and recent orders for customer ID 12345Get real-time stock prices for symbols in my portfolio topicAnalyze sentiment of latest product reviews | | Team productivity | Deploy my microservice to the staging environmentGenerate load test data for the payments serviceCreate a summary dashboard of this week’s incident reports | | Business intelligence | What are the trending products in the last 24 hours?Show revenue impact of the latest feature deploymentGet customer satisfaction scores from support tickets | ## [](#how-it-works)How it works When you start an MCP server, it runs locally on your system and acts as a bridge between AI clients and your data: 1. A user asks their AI agent something like "What’s the weather in London?" 2. The AI client connects to the local MCP server and finds the matching tool 3. The server runs your MCP tools 4. Your tools fetch data, transform it, and return a response 5. The AI agent gets the data and can use it to answer the user ### [](#what-an-mcp-tool-looks-like)What an MCP tool looks like An MCP tool in Redpanda Connect is a YAML file with two parts: the logic (what the tool does) and the metadata (how AI understands it). Here’s a minimal example that returns weather data: ```yaml http: url: "https://wttr.in/${! this.city }?format=j1" verb: GET meta: mcp: enabled: true name: get_weather description: "Get current weather for a city" properties: - name: city type: string description: "City name" required: true ``` When an AI client asks about weather, it calls this tool with the city name. The tool fetches data from the weather API and returns it. ## [](#mcp-specification-support)MCP specification support MCP servers implement the open MCP protocol for tool exposure. Only the tool concept from the MCP server specification is supported. Features such as MCP resources and prompts are not yet available. For full details, see the [official MCP server specification](https://modelcontextprotocol.io/specification/2025-06-18/server). ## [](#next-steps)Next steps Continue your learning journey with these resources: - [MCP Server Quickstart](../quickstart/): Start your first MCP server - [MCP Tool Execution and Components](../concepts/): Learn about execution and component types - [Create an MCP Tool](../create-tool/): Create custom tools - [Model Context Protocol documentation](https://modelcontextprotocol.io/) --- # Page 6: MCP Server Quickstart **URL**: https://docs.redpanda.com/redpanda-connect/ai-agents/mcp-server/quickstart.md --- # MCP Server Quickstart --- title: MCP Server Quickstart latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: mcp-server/quickstart page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: mcp-server/quickstart.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/ai-agents/pages/mcp-server/quickstart.adoc description: Expose Redpanda Connect components as AI-consumable tools using the Model Context Protocol (MCP). page-topic-type: tutorial personas: ai_agent_developer, streaming_developer, evaluator learning-objective-1: Initialize an MCP server project using the Redpanda CLI learning-objective-2: Create a tool with MCP metadata learning-objective-3: Connect the server to Claude Code and invoke the tool page-git-created-date: "2025-09-30" page-git-modified-date: "2026-01-28" --- This quickstart builds a local MCP server that exposes Redpanda Connect components as tools that AI agents can use. It creates a tool that searches Bluesky posts, then asks Claude Code to perform the search using natural language: > Find recent posts on Bluesky mentioning Redpanda. By completing this quickstart, you will be able to: - Initialize an MCP server project using the Redpanda CLI - Create a tool with MCP metadata - Connect the server to Claude Code and invoke the tool > 💡 **TIP** > > For background on how MCP tools work and when to use each component type, see [MCP Tool Execution and Components](../concepts/). ## [](#prerequisites)Prerequisites You’ll need the following to complete this quickstart: - [Redpanda CLI (`rpk`)](../../../../current/get-started/rpk-install/) - [Claude Code](https://docs.anthropic.com/en/docs/claude-code/setup) - [Node.js and npm](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm) (required for running `npx` commands) ## [](#initialize-an-mcp-server-project)Initialize an MCP server project To get started, create a new MCP server project using the Redpanda CLI. 1. Create a directory for your MCP server project: ```bash mkdir redpanda-connect-mcp cd redpanda-connect-mcp ``` 2. Initialize the MCP server project: ```bash rpk connect mcp-server init ``` This command scaffolds your project with the necessary directories and template YAML files for defining tools and observability resources. ├── o11y │ ├── metrics.yaml │ └── tracer.yaml └── resources ├── caches │ └── example-cache.yaml ├── inputs │ └── example-input.yaml ├── outputs │ └── example-output.yaml └── processors └── example-processor.yaml 3. Make sure your Redpanda Connect version is at least 4.66.1: ```bash rpk connect --version ``` If you need to upgrade, see [Upgrade Redpanda Connect](../../../install/rpk/#upgrade). ## [](#create-a-tool-definition)Create a tool definition In this step, you’ll create an MCP tool that searches public [Bluesky](https://bsky.app/) posts using a customizable query. You’ll define this configuration as a tool that Claude can discover and use. Save the following YAML to a file named `bluesky.yaml` and save it in the `resources/processors/` directory: ```yaml label: search-bluesky-posts (1) try: (2) - mutation: | root.limit = root.limit.number(10) root.limit = [ root.limit, 100 ].min() root.limit = [ root.limit, 1 ].max() meta query_string = "q=" + root.query.escape_url_query() + "&limit=%v".format(root.limit) root = "" - http: url: "https://api.bsky.app/xrpc/app.bsky.feed.searchposts?${! @query_string }" verb: GET - mapping: 'root = this.posts' - unarchive: format: json_array meta: tags: [ bluesky ] (3) mcp: enabled: true description: Search public Bluesky posts based on a query string. (4) properties: (5) - name: query type: string required: true description: A Lucene-style query string to search posts. - name: limit type: number description: Number of posts to return (1-100). Defaults to 10. ``` This example defines a tool that searches public Bluesky posts based on a query string. Important parts of the configuration include: | 1 | A unique label for identification. Claude uses this label to discover and call the tool. | | --- | --- | | 2 | A series of processors to build the HTTP request, handle the response, and format the output. | | 3 | A unique tag (bluesky) to control exposure. You can use tags to group related tools or isolate experiments. | | 4 | A clear description for LLMs helps Claude understand what the tool does. For more details, see Anatomy of an MCP tool. | | 5 | Structured properties for inputs like query and limit. | ## [](#start-the-mcp-server)Start the MCP server 1. Navigate to your MCP server project directory if you’re not already there: ```bash cd redpanda-connect-mcp ``` 2. Start the MCP server to expose the tool over HTTP: ```bash rpk connect mcp-server --address localhost:4195 --tag bluesky ``` You should see output like this: time=2025-06-27T15:20:27.976+01:00 level=INFO msg="Registering processor tool" label=search-bluesky-posts time=2025-06-27T15:20:27.978+01:00 level=INFO msg="Successfully loaded Redpanda license" expires\_at=2035-06-25T15:20:27+01:00 license\_org="" license\_type="open source" This command creates an MCP server listening on `localhost:4195`, and makes your tool discoverable to AI agents. Leave this terminal open while you interact with Claude Code. > 💡 **TIP: Limit exposure** > > Only tools with the specified `--tag` are exposed. This helps you: > > - Keep experiments isolated > > - Avoid exposing sensitive functionality accidentally > > - Create sets of tools that are relevant to specific agents or workflows ## [](#connect-claude-code-to-your-mcp-server)Connect Claude Code to your MCP server Now that your MCP server is running and exposing tools, you can connect Claude Code so it can discover and use them. Claude Code communicates with MCP servers using the standard input/output (stdio) transport protocol. Because your Redpanda Connect MCP server exposes tools over HTTP with Server-Sent Events (SSE), you need a bridge to convert between these protocols. The [`mcp-remote` utility](https://www.npmjs.com/package/mcp-remote) provides this bridge. When you connect Claude Code: - `mcp-remote` subscribes to your server’s SSE endpoint (`[http://localhost:4195/sse](http://localhost:4195/sse)`). - It translates HTTP/SSE messages into stdio format that Claude understands. - Claude automatically discovers your tools and their metadata. - You can ask Claude in natural language to use your tools. - Claude calls the appropriate tool based on your request. To set up this connection: 1. Open a new terminal window. 2. To install `mcp-remote`, run: ```bash claude mcp add local -- npx mcp-remote http://localhost:4195/sse ``` You should see output like this: Added stdio MCP server local with command: npx mcp-remote http://localhost:4195/sse to local config 3. Verify that the local input channel is set up correctly by running: ```bash claude /mcp ``` You should see an entry for `local`. 4. Press **Enter** until you see the tools list. Tools for local (1 tools) │ ❯ 1. search-bluesky-posts 5. Press **Esc** until you return to the main prompt. ## [](#use-the-tool-with-claude)Use the tool with Claude Now you can see MCP in action. When you ask Claude to search Bluesky, watch what happens behind the scenes. 1. Enter the following prompt to start a conversation with Claude Code: ```bash Search Bluesky for the latest news about Redpanda Data ``` > 💡 **TIP** > > If you exited the prompt, you can re-enter it by running: > > ```bash > claude Search Bluesky for the latest news about Redpanda Data > ``` 2. When Claude prompts you for permissions, select **Yes**: \> Search Bluesky for the latest news about Redpanda Data ⏺ local:search-bluesky-posts (MCP)(value: "Redpanda") ╭───────────────────────────────────────────── │ Tool use │ local:search-bluesky-posts(value: "Redpanda") (MCP) │ Search public Bluesky posts based on a query string. │ Do you want to proceed? │ ❯ 1. Yes │ 2. Yes, and don't ask again for local:search-bluesky-posts commands in /my-agent │ │ 3. No, and tell Claude what to do differently (esc) This is what happens: - Claude analyzes your natural language request. - Claude identifies the `search-bluesky-posts` tool as the right one to use. - Claude extracts "Redpanda Data" as the query parameter. - Claude calls your local MCP server via `mcp-remote`. - Your Redpanda Connect configuration executes the Bluesky API request. - Results flow back through the MCP server to Claude. - Claude synthesizes the results into a conversational response. Here’s an example of what the result might look like: ⏺ Based on the search results from Bluesky, here are the latest posts about Redpanda Data: Recent Company Activity: - Product Development: Expanding Redpanda Connect beyond data infrastructure to support AI agent tools like Slack, GitHub, and Google Drive - New Snowflake Connector: Launched with 2x faster throughput than Kafka Connect for data streaming Events & Marketing: - Participated in Data+AI Summit (Booth E500) - Hosting disaster recovery streamcast on June 12th - Fireside chat with Entain's Chief Data Officer about replacing Confluent with Redpanda in 2 weeks Success Stories: - Gaming company Entain replaced Confluent with Redpanda in just two weeks - Focus on real-time financial data streaming to Snowflake for analytics The posts show Redpanda positioning itself as a faster, simpler alternative to Kafka/Confluent with strong momentum in hiring and customer wins. > 📝 **NOTE** > > If you change the YAML configuration of your tools, restart the MCP server to pick up the changes. ## [](#stop-or-disconnect-the-mcp-server)Stop or disconnect the MCP server To disconnect or stop the MCP server, press Ctrl+C in the terminal where the server is running. This will gracefully shut down the MCP server process and disconnect any connected clients. You can also close the terminal window or kill the process using standard OS commands (such as `kill ` on Linux/macOS). ## [](#troubleshoot)Troubleshoot If you encounter issues during this quickstart: - **Tool not appearing**: Verify `meta.mcp.enabled: true` is set and the tag matches your `--tag` argument. - **Connection issues**: Ensure the MCP server is running and listening on the expected port. - **JSON schema errors**: Upgrade to at least version 4.66.1 of Redpanda Connect. For detailed solutions, see [Troubleshoot MCP Servers](../troubleshooting/). ## [](#next-steps)Next steps You’ve built a working MCP tool and connected it to Claude Code. Here’s where to go next: - [MCP Tool Execution and Components](../concepts/): Understand how MCP tools differ from pipelines - [Create an MCP Tool](../create-tool/): Build production-quality tools with validation - [MCP Tool Design](../best-practices/): Apply naming and design guidelines - [MCP Tool Patterns](../tool-patterns/): Find reusable patterns - [Troubleshoot MCP Servers](../troubleshooting/): Diagnose common issues --- # Page 7: MCP Tool Patterns **URL**: https://docs.redpanda.com/redpanda-connect/ai-agents/mcp-server/tool-patterns.md --- # MCP Tool Patterns --- title: MCP Tool Patterns latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: mcp-server/tool-patterns page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: mcp-server/tool-patterns.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/ai-agents/pages/mcp-server/tool-patterns.adoc description: Cookbook of reusable patterns for building MCP tools with Redpanda Connect. page-topic-type: cookbook personas: ai_agent_developer, data_engineer learning-objective-1: Find reusable patterns for common MCP tool scenarios learning-objective-2: Apply validation and error handling patterns for production robustness learning-objective-3: Format responses consistently for AI client consumption page-git-created-date: "2026-01-13" page-git-modified-date: "2026-01-13" --- When building tools, use these patterns as starting points for common scenarios. For step-by-step instructions, see [Create an MCP Tool](../create-tool/). For design guidelines, see [MCP Tool Design](../best-practices/). After reading this page, you will be able to: - Find reusable patterns for common MCP tool scenarios - Apply validation and error handling patterns for production robustness - Format responses consistently for AI client consumption ## [](#read-data)Read data Use [inputs](../../../components/inputs/about/) to create tools that read from data sources or generate sample data. ### [](#data-generators)Generate test data **When to use:** Development and testing environments where you need synthetic data, load testing scenarios, or demonstrating data flows without real data sources. **Example use cases:** Mock user events, test order data, synthetic sensor readings, demo data for presentations. ```yaml generate: interval: 1s count: 0 mapping: | root.id = uuid_v4() root.timestamp = now() root.user_id = random_int(min: 1000, max: 9999) root.event_type = ["login", "purchase", "logout"].index(random_int(max: 2)) root.amount = if this.event_type == "purchase" { random_int(min: 10, max: 500) } ``` See also: [`generate` input component](../../../components/inputs/generate/) ### [](#consume-from-redpanda)Consume from Redpanda topics **When to use:** Processing events from Redpanda topics, building event-driven AI agents, consuming audit logs, or subscribing to data change streams. **Example use cases:** Monitor order events, process user activity streams, consume IoT sensor data, react to system notifications. > 📝 **NOTE** > > This example requires setting the `REDPANDA_BROKERS` environment variable with your Redpanda broker addresses. ```yaml label: get_recent_events redpanda: seed_brokers: [ "${REDPANDA_BROKERS}" ] topics: [ "user-events" ] consumer_group: "mcp-event-reader" meta: tags: [ consuming, events, example ] mcp: enabled: true description: "Retrieve recent events from a Redpanda topic. Use the count property to limit results." ``` ### [](#stream-processing)Process streaming data **When to use:** Real-time analytics, windowed aggregations, computing metrics over time, or building streaming dashboards. **Example use cases:** Calculate rolling averages, count events per time window, detect anomalies in streams, aggregate metrics. ```yaml label: stream_analytics redpanda: seed_brokers: [ "${REDPANDA_BROKERS}" ] topics: [ "click-events" ] consumer_group: "mcp-analytics" meta: tags: [ analytics, streaming, example ] mcp: enabled: true description: "Consume click stream data for analytics processing" ``` ## [](#call-external-services)Call external services Use [processors](../../../components/processors/about/) to fetch data from external APIs, databases, or AI services. ### [](#external-api-calls)Call REST APIs **When to use:** Integrating with third-party services, fetching real-time data, calling internal microservices, or enriching event data with external information. **Example use cases:** Fetch user profile from CRM, get product pricing from inventory API, validate addresses with geocoding service, retrieve weather data. ```yaml label: fetch-weather processors: - label: prepare_parameters mutation: | meta city_name = this.city_name - label: fetch_weather http: url: 'https://wttr.in/${! @city_name }?format=j1' verb: GET headers: Accept: "application/json" User-Agent: "redpanda-mcp-server/1.0" - label: format_response mutation: | root = { "city": @city_name, "temperature": this.current_condition.0.temp_C.number(), "feels_like": this.current_condition.0.FeelsLikeC.number(), "humidity": this.current_condition.0.humidity.number(), "pressure": this.current_condition.0.pressure.number(), "description": this.current_condition.0.weatherDesc.0.value, "wind_speed": this.current_condition.0.windspeedKmph.number(), "metadata": { "source": "wttr.in", "fetched_at": now().ts_format("2006-01-02T15:04:05.000Z") } } meta: tags: [ example, weather, api ] mcp: enabled: true description: "Fetch current weather information for a specified city" properties: - name: city_name type: string description: "Name of the city to get weather information for" required: true ``` See also: [`http` processor](../../../components/processors/http/), [`mutation` processor](../../../components/processors/mutation/) ### [](#database-queries)Query databases **When to use:** Retrieving customer records, querying analytics data, looking up configuration values, or joining streaming data with dimensional data from data warehouses. **Example use cases:** Fetch customer details from PostgreSQL, query sales data from BigQuery, retrieve product catalog from MongoDB, look up reference data. > 📝 **NOTE** > > This example requires setting the `DATABASE_URL` environment variable with your PostgreSQL connection string. ```yaml label: user_orders processors: - label: prepare_parameters mutation: | meta user_id = this.user_id.string() meta limit = this.limit.number().catch(10) - label: query_database sql_select: driver: "postgres" dsn: "${DATABASE_URL}" table: "orders" columns: ["id", "total", "status", "created_at"] where: "user_id = ? AND created_at > NOW() - INTERVAL '30 days'" suffix: "ORDER BY created_at DESC LIMIT ?" args_mapping: root = [@user_id, @limit] - label: format_response mutation: | root = { "user_id": @user_id, "orders": this, "total_count": this.length(), "metadata": { "source": "PostgreSQL", "fetched_at": now().ts_format("2006-01-02T15:04:05.000Z") } } meta: tags: [ database, orders, example ] mcp: enabled: true description: "Get recent orders for a user" properties: - name: user_id type: string description: "User ID to fetch orders for" required: true - name: limit type: number description: "Maximum number of orders to return (default: 10)" required: false ``` See also: [`sql_select` processor](../../../components/processors/sql_select/) ### [](#jira-queries)Query Jira issues **When to use:** Fetching tickets by status, checking assignments, finding recent issues, or building AI agents that interact with project management data. **Example use cases:** Get open bugs for a sprint, find issues assigned to a user, list recently updated tickets, search by custom fields. > 📝 **NOTE** > > The `jira` processor requires an Enterprise license. For more information, see [Enterprise Licensing](../../../get-started/licensing/). ```yaml label: search-jira processors: - generate: count: 1 mapping: | root.jql = this.jql root.maxResults = this.max_results.or(50) root.fields = ["key", "summary", "status", "assignee", "priority"] - jira: base_url: "${JIRA_BASE_URL}" username: "${JIRA_USERNAME}" api_token: "${JIRA_API_TOKEN}" meta: mcp: enabled: true description: "Search Jira issues using JQL. Returns matching issues with key, summary, status, assignee, and priority." properties: - name: jql type: string description: "JQL query (e.g., 'project = DOC AND status = Open')" required: true - name: max_results type: number description: "Maximum issues to return (default: 50)" required: false ``` For more patterns including pagination, custom fields, and creating issues via the HTTP processor, see [Work with Jira Issues](../../../cookbooks/jira/). ### [](#ai-llm-integration)Integrate with AI/LLM services **When to use:** Generating embeddings for semantic search, calling LLM APIs for text generation, building RAG pipelines, or analyzing sentiment. **Example use cases:** Generate embeddings for documents, classify customer feedback, summarize long text, extract entities, answer questions with context. #### [](#openai-chat-completion)OpenAI chat completion ```yaml openai_chat_completion: api_key: "${OPENAI_API_KEY}" model: "gpt-4" prompt: | Analyze this customer feedback and provide: 1. Sentiment (positive/negative/neutral) 2. Key themes 3. Actionable insights Feedback: ${! json("feedback_text") } max_tokens: 500 ``` See also: [`openai_chat_completion`](../../../components/processors/openai_chat_completion/), [`openai_embeddings`](../../../components/processors/openai_embeddings/) #### [](#generate-embeddings)Generate embeddings ```yaml openai_embeddings: api_key: "${OPENAI_API_KEY}" model: "text-embedding-3-small" text: ${! json("content") } ``` See also: [`cohere_embeddings`](../../../components/processors/cohere_embeddings/), [`gcp_vertex_ai_embeddings`](../../../components/processors/gcp_vertex_ai_embeddings/) ## [](#write-data)Write data Use [outputs](../../../components/outputs/about/) to write data to Redpanda topics or cache stores. ### [](#publish-to-redpanda)Publish to Redpanda topics **When to use:** Publishing events to Redpanda for consumption by other services, creating event sourcing patterns, building audit trails, or triggering downstream workflows. **Example use cases:** Publish order confirmations, emit audit events, trigger notifications, create event-driven workflows. > 📝 **NOTE** > > This example requires setting the `REDPANDA_BROKERS` environment variable. ```yaml label: publish_order_events redpanda: seed_brokers: [ "${REDPANDA_BROKERS}" ] topic: "order-events" key: "${! this.order_id }" partitioner: "manual" partition: "${! this.customer_id.hash(algorithm:\"xxhash64\") % 8 }" max_in_flight: 10 meta: tags: [ publishing, orders, example ] mcp: enabled: true description: "Publish order events to Redpanda for downstream processing" properties: - name: order_id type: string description: "Unique order identifier" required: true - name: customer_id type: string description: "Customer ID for partitioning" required: true - name: order_data type: string description: "Order details as JSON string (items, total, etc.)" required: true ``` #### [](#outputs-with-processors)Outputs with processors Output tools can include processors to transform data before publishing. This pattern is useful when you need to process data and save the result to a destination in a single tool. **When to use:** Processing user input with an LLM and saving the response, transforming data before publishing to a topic, enriching events before writing to external systems. ```yaml label: summarize_and_store processors: - label: generate_summary openai_chat_completion: api_key: "${OPENAI_API_KEY}" model: "gpt-4" prompt: "Summarize this document in 2-3 sentences: ${! json(\"content\") }" - label: prepare_record mutation: | root = { "original": this.content, "summary": this.choices.0.message.content, "processed_at": now().ts_format("2006-01-02T15:04:05Z") } redpanda: seed_brokers: [ "${REDPANDA_BROKERS}" ] topic: "summaries" meta: tags: [ publishing, llm, example ] mcp: enabled: true description: "Summarize a document using an LLM and store the result in Redpanda" properties: - name: content type: string description: "Document content to summarize" required: true ``` ### [](#caching)Cache data **When to use:** Reducing repeated API calls, storing lookup tables, caching database query results, or maintaining session state across tool invocations. **Example use cases:** Cache user profiles, store API rate limit counters, maintain configuration values, cache product catalogs. ```yaml memory: default_ttl: 300s compaction_interval: 60s ``` See also: [`memory` cache](../../../components/caches/memory/), [`redpanda` output](../../../components/outputs/redpanda/) ## [](#transform-data)Transform data Use Bloblang and processors to transform, validate, and route data. ### [](#data-transformation)Transform and validate **When to use:** Converting data formats, validating schemas, filtering events, enriching messages with computed fields, or normalizing data structures. **Example use cases:** Parse JSON payloads, validate required fields, add timestamps, convert units, mask sensitive data, aggregate nested objects. ```yaml mapping: | # Parse and validate incoming data root.user_id = this.user_id.or(throw("user_id is required")) root.timestamp = now().ts_format("2006-01-02T15:04:05Z07:00") # Transform and enrich root.email_domain = this.email.split("@").index(1) root.is_premium = this.subscription_tier == "premium" # Filter sensitive data root.profile = this.profile.without("ssn", "credit_card") ``` See also: [`mapping` processor](../../../components/processors/mapping/), [Bloblang guide](../../../guides/bloblang/about/) ### [](#event-driven-workflows)Build event-driven workflows **When to use:** Orchestrating multi-step processes, responding to business events, implementing saga patterns, or coordinating microservices. **Example use cases:** Order fulfillment workflows, approval processes, notification cascades, data pipeline orchestration. ```yaml label: event_triggered_workflow redpanda: seed_brokers: [ "${REDPANDA_BROKERS}" ] topics: [ "order-events" ] consumer_group: "workflow-trigger" meta: tags: [ workflows, events, example ] mcp: enabled: true description: "Consume order events to trigger workflows" ``` See also: [`redpanda` input](../../../components/inputs/redpanda/) ## [](#production-readiness)Production readiness Build production-ready tools with proper input validation, error handling, and response formatting. ### [](#input-validation)Validate input AI clients may send unexpected or malformed input. Validate early to return helpful error messages instead of cryptic failures from downstream components. Always validate inputs before processing. This prevents errors and provides clear feedback to the AI client. The following example shows a basic validation pattern: ```yaml - label: validate_input mutation: | let city = this.city.or("").trim() root = if $city == "" { {"error": "City name is required"} } else { {"city": $city} } ``` This validation does three things: First, `.or("")` provides an empty string default if the `city` field is missing, which prevents null errors. Then, `.trim()` removes whitespace so `" "` doesn’t pass as a valid city. Then, the `if` expression returns either an error object or the validated data. The AI client receives clear feedback either way. #### [](#essential-validation-methods)Essential validation methods Use these [Bloblang methods](../../../guides/bloblang/methods/) for input validation: | Method | Purpose | Example | | --- | --- | --- | | .or(default) | Provide fallback for missing fields | this.city.or("unknown") | | .trim() | Remove leading/trailing whitespace | this.name.trim() | | .exists("field") | Check if a field is present | this.exists("email") | | .type() | Get the type of a value | this.count.type() == "number" | | .length() | Check string or array length | this.items.length() > 0 | | .re_match(pattern) | Validate against regex | this.email.re_match("^@[^@]$") | | .number() | Convert and validate as number | this.quantity.number() | #### [](#sanitize-string-inputs)Sanitize string inputs Remove potentially dangerous characters from user inputs. This is especially important when inputs will be used in URLs, database queries, or shell commands: ```yaml - label: sanitize_input mutation: | let clean_city = this.city.or("").trim().re_replace_all("[^a-zA-Z\\s-]", "") root = if $clean_city == "" { {"error": "City name contains only invalid characters"} } else { {"city": $clean_city} } ``` The regex `[^a-zA-Z\\s\\-]` matches any character that is not a letter, space, or hyphen, and `re_replace_all` removes all matches. An input like `"New York!@#$"` becomes `"New York"`. The `meta` keyword stores the result in message metadata (using `@sanitized_city`), keeping it separate from the message body until validation passes. For regex replacement syntax, see [`re_replace_all`](../../../guides/bloblang/methods/#re_replace_all). #### [](#validate-numeric-ranges)Validate numeric ranges Check that numeric inputs fall within acceptable bounds: ```yaml - label: validate_quantity mutation: | let qty = this.quantity.or(0).number() root = if $qty < 1 { {"error": "Quantity must be at least 1", "received": $qty} } else if $qty > 1000 { {"error": "Quantity cannot exceed 1000", "received": $qty} } else { {"quantity": $qty, "valid": true} } ``` This example chains `.or(0)` with `.number()` to handle both missing values and type conversion. The chained `if`/`else if` checks both lower and upper bounds. Including the received value in error responses helps AI clients understand what went wrong and correct their input. #### [](#validate-multiple-fields)Validate multiple fields For forms or complex inputs, collect all errors before returning. This gives AI clients a complete list of problems to fix rather than failing on the first error: ```yaml - label: validate_order mutation: | let errors = [] let errors = if !this.exists("order_id") || this.order_id == "" { $errors.append("order_id is required") } else { $errors } let errors = if !this.exists("items") || this.items.length() == 0 { $errors.append("at least one item is required") } else { $errors } let errors = if this.exists("email") && !this.email.contains("@") { $errors.append("invalid email format") } else { $errors } root = if $errors.length() > 0 { {"valid": false, "errors": $errors} } else { {"valid": true, "order_id": this.order_id} } ``` The pattern uses variable reassignment (`let errors = …​`) to accumulate errors into an array. Each check appends to the array if validation fails, or returns the unchanged array if it passes. At the end, if any errors were collected, the response includes all of them. Notice that the email validation only runs if the field exists. This allows optional fields that, when provided, must be valid. #### [](#validate-enum-values)Validate enum values Restrict inputs to a set of allowed values. This prevents invalid states and provides helpful feedback when the input doesn’t match: ```yaml - label: validate_status mutation: | let allowed = ["pending", "approved", "rejected"] let status = this.status.or("").lowercase() root = if $status == "" { {"error": "status is required", "allowed": $allowed} } else if !$allowed.contains($status) { {"error": "invalid status", "received": $status, "allowed": $allowed} } else { {"status": $status, "valid": true} } ``` The `lowercase()` call normalizes the input so `"PENDING"`, `"Pending"`, and `"pending"` all match. When validation fails, the error response includes the list of allowed values. This helps AI clients self-correct without needing to look up valid options. For more details, see [`contains`](../../../guides/bloblang/methods/#contains) and [`lowercase`](../../../guides/bloblang/methods/#lowercase). #### [](#use-throw-for-validation-failures)Use throw for validation failures Use [`throw()`](../../../guides/bloblang/functions/#throw) to stop processing with an error message. This is useful when validation failure should stop the entire tool execution: ```yaml - label: require_auth mutation: | root = if !this.exists("api_key") || this.api_key == "" { throw("API key is required for this operation") } else { this } ``` Unlike returning an error object, `throw()` immediately stops the processor chain and triggers any `catch` block that follows. Use `throw()` for critical validation failures where continuing would be pointless or dangerous. The `else` branch returns `this` unchanged, passing all input fields to the next processor. ### [](#error-handling)Handle errors External services fail. Databases go down. APIs return unexpected responses. Wrap risky operations in error handling so your tool returns useful error messages instead of crashing. Wrap operations that can fail in `try`/`catch` blocks. This ensures the tool returns useful errors instead of failing silently. ```yaml processors: - try: - http: url: "https://api.example.com/data" verb: GET - catch: - mutation: | root.error = true root.message = "Request failed: " + error() ``` For full configuration options, see [`try` processor](../../../components/processors/try/) and [`catch` processor](../../../components/processors/catch/). #### [](#return-error-details)Return error details The [`error()`](../../../guides/bloblang/functions/#error) function returns the error message from the most recent failure. Use it in `catch` blocks to capture what went wrong: ```yaml - label: handle_errors catch: - mutation: | root = { "success": false, "error_message": error(), "timestamp": now().format_timestamp("2006-01-02T15:04:05Z07:00") } ``` #### [](#set-timeouts)Set timeouts Always set explicit timeouts on external calls to prevent tools from hanging indefinitely: ```yaml - label: fetch_with_timeout try: - http: url: "https://httpbin.org/get" verb: GET timeout: "10s" # Fail after 10 seconds retries: 2 # Retry twice before failing retry_period: "1s" ``` For all timeout and retry options, see [`http` processor](../../../components/processors/http/). #### [](#handle-specific-error-types)Handle specific error types Create different responses based on error type: ```yaml - catch: - mutation: | let err = error() root.error = true root.error_type = if $err.contains("timeout") { "TIMEOUT" } else if $err.contains("connection refused") { "CONNECTION_ERROR" } else if $err.contains("404") { "NOT_FOUND" } else { "UNKNOWN" } root.message = $err root.retry_suggested = root.error_type == "TIMEOUT" || root.error_type == "CONNECTION_ERROR" ``` #### [](#log-errors-for-debugging)Log errors for debugging Add logging inside `catch` blocks to aid troubleshooting: ```yaml - catch: - log: message: "Tool failed: ${! error() }" level: ERROR fields_mapping: | root.input = this root.error = error() - mutation: | root.error = true root.message = error() ``` For log level options, see [`log` processor](../../../components/processors/log/). #### [](#preserve-input-context-in-errors)Preserve input context in errors Include original input data in error responses to help AI clients retry with corrections: ```yaml - label: validate_and_fetch try: - mutation: | meta original_input = this - mutation: | root = throw("User not found in database") - label: handle_errors catch: - mutation: | root = { "error": "Failed to fetch user", "details": error(), "input_received": @original_input, "suggestion": "Verify the user_id exists" } ``` ### [](#response-formatting)Format responses AI clients work best with clean, predictable response structures. Transform raw component output into consistent formats. Structure responses consistently so AI clients can interpret them reliably. The following example takes a raw weather API response and transforms it into a clean, predictable format: ```yaml - label: format_response mapping: | root = { "city": this.location.name, "temperature_c": this.current.temp_c.number(), "description": this.current.condition.text, "timestamp": now().ts_format("2006-01-02T15:04:05Z") } ``` This mapping does four things: - Extracts the city name from a nested `location.name` field - Converts `temp_c` to a number type (APIs sometimes return numbers as strings) - Pulls out the weather description text - Adds a timestamp so the AI client knows when the data was fetched The result is a flat JSON object with predictable field names and types, rather than the raw API response which might have deeply nested structures or inconsistent formatting. #### [](#type-coercion-methods)Type coercion methods Use [type coercion methods](../../../guides/bloblang/methods/) to ensure fields have the correct data types: | Method | Purpose | Example | | --- | --- | --- | | .string() | Convert to string | this.id.string() becomes "123" | | .number() | Convert to number | this.price.number() becomes 19.99 | | .bool() | Convert to boolean | this.active.bool() becomes true | | .int64() | Convert to 64-bit integer | this.count.int64() becomes 42 | | .float64() | Convert to 64-bit float | this.ratio.float64() becomes 0.75 | #### [](#format-timestamps)Format timestamps Use [`now()`](../../../guides/bloblang/functions/#now) with [`format_timestamp()`](../../../guides/bloblang/methods/#format_timestamp) for consistent time formatting: ```yaml - label: add_timestamps mutation: | root = this root.timestamp = now().format_timestamp("2006-01-02T15:04:05Z07:00") root.date = now().format_timestamp("2006-01-02") root.time = now().format_timestamp("15:04:05") ``` This example preserves all existing fields (`root = this`) and adds three timestamp fields. The `now()` function returns the current time, and `format_timestamp()` converts it to a string. Each field uses a different format: full ISO 8601 timestamp, date only, and time only. The format string uses Go’s reference time layout. Common patterns: | Format | Output example | | --- | --- | | "2006-01-02T15:04:05Z07:00" | 2024-03-15T14:30:00-07:00 (ISO 8601) | | "2006-01-02" | 2024-03-15 | | "15:04:05" | 14:30:00 | | "Mon, 02 Jan 2006" | Fri, 15 Mar 2024 | #### [](#extract-nested-fields)Extract nested fields API responses often have deeply nested structures. Extract only the fields your AI client needs and flatten them into a simple object: ```yaml - label: extract_user_data mapping: | root = { "user_id": this.data.user.id.string(), "name": this.data.user.profile.display_name, "email": this.data.user.contact.email, "is_verified": this.data.user.status.verified.bool(), "created_at": this.data.user.metadata.created_at } ``` This mapping navigates a nested structure like `{"data": {"user": {"id": 123, "profile": {"display_name": "…​"}}}}` and creates a flat response. The dot notation (`this.data.user.id`) drills down through nested objects. Type coercion (`.string()`, `.bool()`) ensures consistent output types. For navigating nested structures, see [dot notation](../../../guides/bloblang/about/#dot-notation). #### [](#handle-arrays)Handle arrays When your data contains arrays, you can transform each element, extract specific items, or compute aggregates: ```yaml - label: format_items mapping: | root = { "total_items": this.items.length(), "items": this.items.map_each(item -> { "id": item.id.string(), "name": item.name, "price": item.price.number() }), "first_item": this.items.index(0).name, "item_names": this.items.map_each(i -> i.name) } ``` This example demonstrates four array operations: - `length()` returns the array size for the `total_items` count - `map_each()` transforms each item into a new object with only the fields you need - `index(0)` accesses the first element (zero-indexed) to get the first item’s name - A second `map_each()` extracts just the names into a simple string array For array operations, see [`map_each()`](../../../guides/bloblang/methods/#map_each) and [`index()`](../../../guides/bloblang/methods/#index). #### [](#include-fields-conditionally)Include fields conditionally Sometimes you want to include fields only when they have meaningful values. This avoids returning `null` or empty fields that clutter the response: ```yaml - mutation: | root.id = this.id root.name = this.name root.email = if this.exists("email") && this.email != "" { this.email } else { deleted() } root.phone = if this.exists("phone") { this.phone } else { deleted() } ``` This mapping starts with required fields (`id`, `name`), then conditionally adds optional fields. The `exists()` check prevents errors when accessing missing fields. When the condition is false, `deleted()` removes the field entirely from the output. The AI client won’t see `"email": null`. The field simply won’t exist. The [`deleted()`](../../../guides/bloblang/functions/#deleted) function removes the field from the output entirely. #### [](#filter-sensitive-data)Filter sensitive data When your data source contains sensitive fields, strip them before returning responses to the AI client: ```yaml - mutation: | root = this.without("password", "ssn", "api_key", "internal_notes") ``` The `without()` method creates a copy of the object with the specified fields removed. This is safer than manually selecting fields because new fields added to the source data are included automatically, so you only need to maintain the exclusion list. Use this when returning database records or API responses that might contain credentials or personal information. For field removal, see [`without()`](../../../guides/bloblang/methods/#without). #### [](#wrap-responses-in-a-success-envelope)Wrap responses in a success envelope When AI clients call multiple tools, they need a predictable way to check if the call succeeded. Wrapping responses in a consistent envelope structure makes this easy: ```yaml - label: format_success mapping: | root = { "success": true, "data": { "user_id": this.id, "name": this.name }, "timestamp": now().format_timestamp("2006-01-02T15:04:05Z07:00") } ``` Both success and error responses share the same top-level structure: a `success` boolean, a payload field (`data` or `error`), and a `timestamp`. The AI client can check `success` first, then access the appropriate field. The error response uses the `catch` processor to handle failures and the `error()` function to capture the error message. ## [](#advanced-workflows)Advanced workflows Build multi-step workflows with dynamic configuration, conditional logic, and observability. ### [](#dynamic-configuration)Dynamic configuration Build tools that adapt their behavior based on input parameters: ```yaml processors: - label: dynamic_config mutation: | # Choose data source based on environment meta env = this.environment | "production" meta table_name = match @env { "dev" => "dev_orders", "staging" => "staging_orders", "production" => "prod_orders", _ => "dev_orders" } # Adjust query complexity based on urgency meta columns = if this.detailed.bool().catch(false) { ["order_id", "customer_id", "total", "items", "shipping_address"] } else { ["order_id", "customer_id", "total"] } ``` ### [](#conditional-processing)Conditional processing Build tools that branch based on input or data characteristics: ```yaml processors: - label: conditional_processing switch: - check: this.data_type == "json" processors: - mapping: | root.parsed_data = this.content.parse_json() root.format = "json" - check: this.data_type == "csv" processors: - mapping: | root.parsed_data = this.content.parse_csv() root.format = "csv" - processors: - mapping: | root.error = "Unsupported data type" root.supported_types = ["json", "csv"] ``` ### [](#secrets)Secrets and credentials Securely handle multiple credentials and API keys using environment variables. Here is an example of using an API key from environment variables. 1. Set an environment variable with your API key: ```bash export EXTERNAL_API_KEY="your-api-key-here" ``` 2. Reference the environment variable in your configuration: ```yaml processors: - label: call_external_api http: url: "https://api.example.com/data" verb: GET headers: Authorization: "Bearer ${EXTERNAL_API_KEY}" (1) Accept: "application/json" ``` | 1 | The environment variable is injected at runtime. Never store the actual API key in your YAML. The actual secret value never appears in your configuration files or logs. | | --- | --- | ### [](#monitoring-debugging-and-observability)Monitoring, debugging, and observability Use structured logging, request tracing, and performance metrics to gain insights into tool execution. ```yaml label: observable_tool processors: - label: init_tracing mutation: | # Generate correlation ID for request tracing meta req_id = uuid_v7() meta start_time = now() # Log request start with structured data root.trace = { "request_id": @req_id, "timestamp": @start_time.ts_format("2006-01-02T15:04:05.000Z"), "tool": "observable_tool", "version": "1.0.0" } - label: log_request_start log: message: "MCP tool request started" fields: request_id: "${! @req_id }" tool_name: "observable_tool" input_params: "${! this.without(\"trace\") }" user_agent: "${! meta(\"User-Agent\").catch(\"unknown\") }" level: "INFO" - label: finalize_response mutation: | # Calculate total execution time meta duration = (now().ts_unix_nano() - @start_time.ts_unix_nano()) / 1000000 # Add trace information to response root.metadata = { "request_id": @req_id, "execution_time_ms": @duration, "timestamp": now().ts_format("2006-01-02T15:04:05.000Z"), "tool": "observable_tool", "success": !this.exists("error") } - label: log_completion log: message: "MCP tool request completed" fields: request_id: "${! @req_id }" duration_ms: "${! this.metadata.execution_time_ms }" success: "${! this.metadata.success }" result_size: "${! content().length() }" level: "INFO" meta: tags: [ example ] mcp: enabled: true description: "Example tool with comprehensive observability and error handling" properties: - name: user_id type: string description: "User ID to fetch data for" required: true ``` Observability features: - **Correlation IDs**: Use `uuid_v7()` to generate unique request identifiers for tracing - **Execution timing**: Track how long your tools take to execute using nanosecond precision - **Structured logging**: Include consistent fields like `request_id`, `duration_ms`, `tool_name` - **Request/response metadata**: Log input parameters and response characteristics - **Success tracking**: Monitor whether operations complete successfully You can test this pattern by invoking the tool with valid and invalid parameters, and observe the structured logs for tracing execution flow. For example, with a user ID of 1, you might see logs like: ```json { "metadata": { "execution_time_ms": 0.158977, "request_id": "019951ab-d07d-703f-aaae-7e1c9a5afa95", "success": true, "timestamp": "2025-09-16T08:37:18.589Z", "tool": "observable_tool" }, "trace": { "request_id": "019951ab-d07d-703f-aaae-7e1c9a5afa95", "timestamp": "2025-09-16T08:37:18.589Z", "tool": "observable_tool", "version": "1.0.0" }, "user_id": "1" } ``` See also: [`log` processor](../../../components/processors/log/), [`try` processor](../../../components/processors/try/), [Bloblang functions](../../../guides/bloblang/functions/) (for timing and ID generation) ### [](#multi-step-data-enrichment)Multi-step data enrichment Build tools that combine data from multiple sources. This workflow fetches customer data from a SQL database, enriches it with recent order history, and computes summary metrics. ```yaml label: customer_enrichment processors: - label: fetch_customer_base branch: processors: - sql_select: driver: "postgres" dsn: "${POSTGRES_DSN}" table: "customers" where: "customer_id = ?" args_mapping: 'root = [this.customer_id]' result_map: 'root.customers = this' - label: enrich_with_orders branch: processors: - sql_select: driver: "postgres" dsn: "${POSTGRES_DSN}" table: "orders" where: "customer_id = ? AND created_at >= NOW() - INTERVAL ''30 days''" args_mapping: 'root = [this.customer_id]' result_map: 'root.orders = this' - label: combine_data mutation: | let order_totals = this.orders.map_each(o -> o.total) root = { "customer": this.customers.index(0), "recent_orders": this.orders, "metrics": { "total_orders": this.orders.length(), "total_spent": $order_totals.sum(), "avg_order_value": if $order_totals.length() > 0 { $order_totals.sum() / $order_totals.length() } else { 0 } } } meta: tags: [ example ] mcp: enabled: true description: "Get comprehensive customer profile with recent order history and metrics" properties: - name: customer_id type: string description: "Customer ID to analyze" required: true ``` See also: [`sql_select` processor](../../../components/processors/sql_select/), [Bloblang functions](../../../guides/bloblang/about/) (for data manipulation and aggregations) ### [](#workflow-orchestration)Workflow orchestration Coordinate complex workflows with multiple steps and conditional logic. This workflow simulates a complete order processing pipeline with mock data for inventory and processing tiers. This allows you to test the full logic without needing real external systems. ```yaml label: order_workflow processors: - label: validate_order mutation: | # Validation logic root = if this.total <= 0 { throw("Invalid order total") } else { this } - label: mock_inventory_check mutation: | # Mock inventory data for testing let inventory = { "widget-001": {"quantity": 100, "name": "Standard Widget"}, "widget-premium": {"quantity": 25, "name": "Premium Widget"}, "widget-limited": {"quantity": 2, "name": "Limited Edition Widget"} } let product = $inventory.get(this.product_id) root = if $product == null { throw("Product not found: " + this.product_id) } else if $product.quantity < this.quantity { throw("Insufficient inventory. Available: " + $product.quantity.string()) } else { this.merge({ "inventory_check": "passed", "available_quantity": $product.quantity, "product_name": $product.name }) } - label: route_by_priority switch: - check: 'this.total > 1000' processors: - label: mock_high_value_processing mutation: | # Mock premium processing root = this.merge({ "processing_tier": "premium", "processing_time_estimate": "2-4 hours", "assigned_rep": "premium-team@company.com", "priority_score": 95 }) - check: 'this.customer_tier == "vip"' processors: - label: mock_vip_processing mutation: | # Mock VIP processing root = this.merge({ "processing_tier": "vip", "processing_time_estimate": "1-2 hours", "assigned_rep": "vip-team@company.com", "priority_score": 90, "perks": ["expedited_shipping", "white_glove_service"] }) - processors: - label: mock_standard_processing mutation: | # Mock standard processing root = this.merge({ "processing_tier": "standard", "processing_time_estimate": "24-48 hours", "assigned_rep": "support@company.com", "priority_score": 50 }) - label: finalize_order mutation: | # Add final processing metadata # Calculate estimated fulfillment by parsing processing time let max_hours = this.processing_time_estimate.split("-").index(1).split(" ").index(0).number() root = this.merge({ "order_status": "processed", "processed_at": now().ts_format("2006-01-02T15:04:05.000Z"), "estimated_fulfillment": "TBD - calculated based on processing tier", "processing_time_hours": $max_hours }) meta: tags: [ example ] mcp: enabled: true description: "Process orders with validation, inventory check, and tiered routing (with mocks for testing)" properties: - name: order_id type: string description: "Unique order identifier" required: true - name: product_id type: string description: "Product ID (try: widget-001, widget-premium, widget-limited)" required: true - name: quantity type: number description: "Quantity to order" required: true - name: total type: number description: "Order total in dollars" required: true - name: customer_tier type: string description: "Customer tier (optional: vip, standard)" required: false ``` For the input `{"order_id": "ORD001", "product_id": "widget-001", "quantity": 5, "total": 250, "customer_tier": "vip"}`, the workflow produces: ```json { "assigned_rep": "vip-team@company.com", "available_quantity": 100, "customer_tier": "vip", "estimated_fulfillment": "TBD - calculated based on processing tier", "inventory_check": "passed", "order_id": "ORD001", "order_status": "processed", "perks": [ "expedited_shipping", "white_glove_service" ], "priority_score": 90, "processed_at": "2025-09-16T09:05:29.138Z", "processing_tier": "vip", "processing_time_estimate": "1-2 hours", "processing_time_hours": 2, "product_id": "widget-001", "product_name": "Standard Widget", "quantity": 5, "total": 250 } ``` Notice how the workflow: 1. Preserves original input: `order_id`, `product_id`, `quantity`, `total`, and `customer_tier` pass through unchanged. 2. Adds inventory data: `available_quantity`, `product_name`, and `inventory_check` status from the mock lookup. 3. Routes by customer tier: Since `customer_tier` is `vip`, it gets VIP processing with special `perks` and priority. 4. Enriches with processing metadata: `assigned_rep`, `priority_score`, `processing_tier`, and time estimates. 5. Finalizes with timestamps: `order_status`, `processed_at`, and calculated `processing_time_hours`. > 📝 **NOTE** > > The multi-step enrichment example requires setting the `POSTGRES_DSN` environment variable with your PostgreSQL connection string. ## [](#next-steps)Next steps - [Create an MCP Tool](../create-tool/): Step-by-step tool creation guide - [MCP Tool Design](../best-practices/): Apply naming and design guidelines - [Troubleshoot MCP Servers](../troubleshooting/): Diagnose common issues --- # Page 8: Troubleshoot MCP Servers **URL**: https://docs.redpanda.com/redpanda-connect/ai-agents/mcp-server/troubleshooting.md --- # Troubleshoot MCP Servers --- title: Troubleshoot MCP Servers latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: mcp-server/troubleshooting page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: mcp-server/troubleshooting.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/ai-agents/pages/mcp-server/troubleshooting.adoc description: Diagnose and fix common issues when building and running MCP servers with Redpanda Connect. page-topic-type: troubleshooting personas: ai_agent_developer, streaming_developer, platform_operator learning-objective-1: Diagnose and fix lint and YAML configuration errors learning-objective-2: Resolve runtime issues when tools don't appear or return unexpected results learning-objective-3: Debug client connection problems page-git-created-date: "2026-01-13" page-git-modified-date: "2026-01-13" --- This page helps you diagnose and fix common issues when building and running MCP servers. Use this page to: - Diagnose and fix lint and YAML configuration errors - Resolve runtime issues when tools don’t appear or return unexpected results - Debug client connection problems ## [](#lint-errors)Lint errors Always lint your configuration before starting the server: ```bash rpk connect mcp-server lint ``` ### [](#common-lint-errors)Common lint errors - `unable to infer component type`: Your file contains multiple component types or uses wrapper blocks. Each YAML file must contain only a single component type and should not be wrapped in an `input:` or `output:` block. See [Unable to infer component type](#fix-unable-to-infer). - `unknown field`: A configuration field is misspelled. Check the field name against the component documentation. - `missing required field`: A required field is missing from your configuration. Add the missing field. ### [](#fix-unable-to-infer)Unable to infer component type If you see errors like the following, your YAML file contains more than one component type or uses a wrapper: ```none resources/inputs/redpanda-consume.yaml(1,1) unable to infer component type: [input processors cache_resources meta] resources/outputs/redpanda-publish.yaml(1,1) unable to infer component type: [processors output meta] ``` To fix this, split out each component type into its own file. Incorrect: Multiple component types ```yaml label: incorrect-example input: redpanda: { ... } processors: - mutation: { ... } output: redpanda: { ... } ``` Correct: Single component type ```yaml label: event-reader redpanda: seed_brokers: [ "${REDPANDA_BROKERS}" ] topics: [ "events" ] consumer_group: "mcp-reader" meta: mcp: enabled: true description: "Consume events from Redpanda" ``` ### [](#json-schema-errors)JSON schema errors JSON schema errors indicate that you’re using an outdated version of Redpanda Connect with an incompatible JSON schema format: ```json { "type": "error", "error": { "type": "invalid_request_error", "message": "tools.17.custom.input_schema: JSON schema is invalid..." } } ``` Upgrade to at least version 4.66.1 of Redpanda Connect: ```bash rpk connect --version ``` If you need to upgrade, see [Upgrade Redpanda Connect](../../../install/rpk/#upgrade). ## [](#runtime-issues)Runtime issues ### [](#tool-not-appearing-in-mcp-client)Tool not appearing in MCP client If your tool doesn’t appear in the MCP client’s tool list: 1. Verify that `meta.mcp.enabled: true` is set in your YAML configuration. 2. Check the tool has the correct tag: - Ensure the tool’s `meta.tags` array includes the tag you specified when starting the server. - Example: If you started with `--tag bluesky`, your tool needs `tags: [ bluesky ]`. 3. Verify correct directory structure: - Processors: `resources/processors/` - Inputs: `resources/inputs/` - Outputs: `resources/outputs/` - Caches: `resources/caches/` Example correct structure ```yaml label: my-tool # ... component configuration ... meta: tags: [ my-tag ] # Must match --tag argument mcp: enabled: true # Required for exposure description: Tool description ``` ### [](#tool-returns-unexpected-results)Tool returns unexpected results If your tool runs but returns unexpected data: 1. Check input validation. Add logging to see what inputs the tool receives: ```yaml - log: message: "Received input: ${! json() }" level: DEBUG ``` 2. Verify data transformations. Log intermediate results between processors. 3. Check external API responses. The API may return different data than expected. ## [](#connection-issues)Connection issues ### [](#mcp-client-cant-connect-to-server)MCP client can’t connect to server If your MCP client can’t connect to your local MCP server: 1. Verify the server is running: - Check that the MCP server process is still running in your terminal. - Look for "Registering processor tool" log messages at startup. - Confirm the server is listening on the expected address (default: `localhost:4195`). 2. Check for port conflicts: If port 4195 is already in use, specify a different port: ```bash rpk connect mcp-server --address localhost:4196 --tag my-tag ``` Then update your `mcp-remote` connection: ```bash claude mcp add local -- npx mcp-remote http://localhost:4196/sse ``` 3. Verify the SSE endpoint: - The MCP client connects to `/sse` endpoint (for example, `[http://localhost:4195/sse](http://localhost:4195/sse)`). - Test the endpoint directly: `curl [http://localhost:4195/sse](http://localhost:4195/sse)` ### [](#connection-drops-or-times-out)Connection drops or times out If connections are unstable: 1. Check network connectivity between the client and server. 2. Verify no firewall rules are blocking the connection. 3. Ensure the MCP server process isn’t being killed by the OS (check system logs). ## [](#debugging)Debugging techniques Use these techniques to systematically isolate and fix issues with your MCP tools. ### [](#add-temporary-logging)Add temporary logging Insert [`log` processors](../../../components/processors/log/) to debug data flow: ```yaml processors: - log: message: "Input received: ${! json() }" level: DEBUG - # ... your processing logic ... - log: message: "Output produced: ${! json() }" level: DEBUG ``` The `${! json() }` syntax uses [Bloblang interpolation](../../../guides/bloblang/functions/#json) to insert the current message content. Remove debug processors before deploying to production. ### [](#test-your-tools)Test your tools Build confidence by testing at each stage: 1. Lint your configuration with `rpk connect mcp-server lint`. 2. Start the server and test with cURL or an MCP client. 3. Connect to your AI client and verify the tool appears. 4. Test end-to-end with realistic prompts. For the complete testing workflow with cURL scripts, see [Test the tool](../create-tool/#_test_the_tool). ### [](#isolate-the-problem)Isolate the problem When debugging complex tools: 1. Test each processor individually by commenting out others. 2. Use static test data instead of live API calls. 3. Check if the issue is in input validation, processing logic, or output formatting. 4. Compare working tools with broken ones to identify differences. ## [](#next-steps)Next steps If you’re still experiencing issues: - [Create an MCP Tool](../create-tool/): Review YAML structure rules - [MCP Tool Execution and Components](../concepts/): Review component type selection --- # Page 9: Components Catalog **URL**: https://docs.redpanda.com/redpanda-connect/components/about.md --- # Components Catalog --- title: Components Catalog latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: about page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/about.adoc description: A searchable catalog of available Redpanda Connect components. page-git-created-date: "2024-05-24" page-git-modified-date: "2025-08-08" --- Use the following table to search for available inputs, outputs, and processors. For a description of the support levels, see [Support levels](#support-levels). Type: All Types Selected ▼ Processor Input Output Scanner Metric Cache Tracer Rate limit Buffer Support: All Support Levels Selected ▼ Certified Community Available in Cloud: All Options Selected ▼ Yes No Enterprise License: All Options Selected ▼ Yes No | Name | Connector Type | Support Level | Enterprise Licensed | Available in Cloud | | --- | --- | --- | --- | --- | | a2a_message | Processor | Certified | Yes | Yes | | amqp_0_9RabbitMQ AMQP | Input, Output | Certified | No | Yes | | amqp_1RabbitMQ AMQP Apache Qpid | Input, Output | Community | No | No | | archiveZIP TAR GZIP | Processor | Certified | No | Yes | | avro | Processor, Scanner | Community | No | Yes | | awk | Processor | Community | No | No | | aws_bedrock_chatAmazon AWS Bedrock Chat | Processor | Certified | No | Yes | | aws_bedrock_embeddingsAmazon AWS Bedrock Embeddings | Processor | Certified | No | Yes | | aws_cloudwatch | Metric | Community | No | No | | aws_cloudwatch_logsAWS CloudWatch Logs Amazon CloudWatch Logs | Input | Community | No | Yes | | aws_dynamodbAWS DynamoDB Amazon DynamoDB DynamoDB | Cache, Output | Community | No | Yes | | aws_dynamodb_cdcAmazon DynamoDB CDC | Input | Certified | Yes | Yes | | aws_dynamodb_partiqlAmazon AWS DynamoDB PartiQL | Processor | Certified | No | Yes | | aws_kinesisAWS Kinesis Amazon Kinesis Kinesis | Input, Output | Certified | No | Yes | | aws_kinesis_firehoseAWS Kinesis Firehose Amazon Kinesis Firehose Kinesis Firehose | Output | Certified | No | Yes | | aws_lambdaAWS Lambda Amazon Lambda Lambda | Processor | Certified | No | Yes | | aws_s3AWS S3 Amazon S3 S3 Simple Storage Service | Cache, Input, Output | Certified | No | Yes | | aws_snsAWS SNS Amazon SNS SNS Simple Notification Service | Output | Community | No | Yes | | aws_sqsAWS SQS Amazon SQS SQS Simple Queue Service | Input, Output | Certified | No | Yes | | azure_blob_storageAzure Blob Storage Microsoft Azure Storage | Input, Output | Certified | No | Yes | | azure_cosmosdbMicrosoft Azure Azure | Input, Output, Processor | Certified | No | Yes | | azure_data_lake_gen2Microsoft Azure Azure | Output | Certified | No | Yes | | azure_queue_storageAzure Queue Storage Microsoft Azure Queue | Input, Output | Certified | No | Yes | | azure_table_storageAzure Table Storage Microsoft Azure Table | Input, Output | Certified | No | Yes | | batched | Input | Certified | No | Yes | | beanstalkd | Input, Output | Community | No | No | | benchmark | Processor | Certified | No | Yes | | bloblang | Processor | Certified | No | Yes | | bounds_check | Processor | Certified | No | Yes | | branch | Processor | Certified | No | Yes | | broker | Input, Output | Certified | No | Yes | | cache | Output, Processor | Certified | No | Yes | | cached | Processor | Certified | No | Yes | | cassandraApache Cassandra | Input, Output | Community | No | No | | catch | Processor | Certified | No | Yes | | chunker | Scanner | Certified | No | Yes | | cockroachdb_changefeed | Input | Community | No | No | | cohere_chat | Processor | Certified | No | Yes | | cohere_embeddings | Processor | Certified | No | Yes | | cohere_rerank | Processor | Certified | No | Yes | | command | Processor | Certified | No | No | | compress | Processor | Certified | No | Yes | | couchbase | Cache, Output, Processor | Community | No | No | | crash | Processor | Certified | No | No | | csvComma-Separated Values | Input, Scanner | Certified | No | Yes | | cyborgdb | Output | Community | No | Yes | | cypher | Output | Community | No | No | | decompress | Processor, Scanner | Certified | No | Yes | | dedupe | Processor | Certified | No | Yes | | discord | Input, Output | Community | No | No | | drop | Output | Certified | No | Yes | | drop_on | Output | Certified | No | Yes | | dynamic | Input, Output | Community | No | No | | elasticsearch_v8 | Output | Certified | No | Yes | | elasticsearch_v9 | Output | Community | No | No | | fallback | Output | Certified | No | Yes | | ffiForeign Function Interface | Processor | Certified | No | No | | file | Cache, Input, Output | Certified | No | No | | for_each | Processor | Certified | No | Yes | | gateway | Input | Certified | Yes | Yes | | gcp_bigqueryGCP BigQuery Google BigQuery BigQuery | Output | Certified | No | Yes | | gcp_bigquery_selectGCP BigQuery Google Cloud GCP | Input, Processor | Certified | No | Yes | | gcp_cloud_storageGCP Cloud Storage Google Cloud Storage GCS | Cache, Input, Output | Certified | No | Yes | | gcp_cloudtraceGCP Cloud Trace | Tracer | Certified | No | Yes | | gcp_pubsubGCP PubSub Google Cloud Pub/Sub GCP Pub/Sub Google Pub/Sub | Input, Output | Certified | No | Yes | | gcp_spanner_cdcGoogle Cloud GCP | Input | Certified | Yes | Yes | | gcp_vertex_ai_chatGCP Vertex AI Google Cloud GCP | Processor | Certified | No | Yes | | gcp_vertex_ai_embeddingsGoogle Cloud GCP | Processor | Certified | No | Yes | | generate | Input | Certified | No | Yes | | git | Input | Certified | No | Yes | | google_drive_download | Processor | Certified | Yes | Yes | | google_drive_list_labels | Processor | Certified | Yes | Yes | | google_drive_search | Processor | Certified | Yes | Yes | | grok | Processor | Community | No | No | | group_by | Processor | Certified | No | Yes | | group_by_value | Processor | Certified | No | Yes | | hdfs | Input, Output | Community | No | No | | http | Processor | Certified | No | Yes | | http_clientHTTP REST API REST | Input, Output | Certified | No | Yes | | http_serverHTTP REST API REST Gateway | Input, Output | Certified | No | Yes | | icebergApache Iceberg Apache Polaris AWS Glue Databricks Unity Catalog | Output | Certified | Yes | Yes | | influxdb | Metric | Community | No | No | | inproc | Input, Output | Certified | No | Yes | | insert_part | Processor | Certified | No | Yes | | jaeger | Tracer | Community | No | No | | javascript | Processor | Certified | No | No | | jira | Processor | Certified | No | Yes | | jmespath | Processor | Certified | No | Yes | | jq | Processor | Certified | No | Yes | | json_api | Metric | Certified | No | No | | json_array | Scanner | Community | No | Yes | | json_documents | Scanner | Certified | No | Yes | | json_schemaJSON Schema | Processor | Certified | No | Yes | | kafkaApache Kafka | Input, Output | Certified | No | Yes | | kafka_franzApache Kafka Kafka | Input, Output | Certified | No | Yes | | lines | Scanner | Certified | No | Yes | | local | Rate_limit | Certified | No | Yes | | log | Processor | Certified | No | Yes | | logger | Metric | Certified | No | No | | lru | Cache | Community | No | Yes | | mapping | Processor | Certified | No | Yes | | memcached | Cache | Community | No | Yes | | memory | Buffer, Cache | Certified | No | Yes | | metric | Processor | Certified | No | Yes | | microsoft_sql_server_cdc | Input | Certified | Yes | Yes | | mongodbMongo | Cache, Input, Output, Processor | Certified | No | Yes | | mongodb_cdcMongoDB CDC | Input | Certified | Yes | Yes | | mqtt | Input, Output | Certified | No | Yes | | msgpack | Processor | Community | No | No | | multilevel | Cache | Certified | No | Yes | | mutation | Processor | Certified | No | Yes | | mysql_cdc | Input | Certified | Yes | Yes | | nanomsg | Input, Output | Community | No | No | | natsNATS.io | Input, Output | Certified | No | Yes | | nats_jetstreamNATS JetStream NATS | Input, Output | Certified | No | Yes | | nats_kvNATS KV | Cache, Input, Output, Processor | Certified | No | Yes | | nats_request_replyNATS Request Reply | Processor | Certified | No | Yes | | nats_streamNATS Stream NATS Streaming NATS | Input, Output | Community | No | No | | none | Buffer, Metric, Tracer | Certified | No | Yes | | noop | Cache, Processor | Certified | No | Yes | | nsq | Input, Output | Community | No | No | | ockam_kafka | Input, Output | Community | No | No | | ollama_chat | Processor | Certified | No | No | | ollama_embeddings | Processor | Certified | No | No | | ollama_moderation | Processor | Certified | No | No | | open_telemetry_collector | Tracer | Community | No | No | | openai_chat_completion | Processor | Certified | No | Yes | | openai_embeddings | Processor | Certified | No | Yes | | openai_image_generation | Processor | Certified | No | Yes | | openai_speech | Processor | Certified | No | Yes | | openai_transcription | Processor | Certified | No | Yes | | openai_translation | Processor | Certified | No | Yes | | opensearch | Output | Certified | No | Yes | | oracledb_cdcOracle CDC OracleDB CDC Oracle Database CDC | Input | Certified | Yes | Yes | | otlp_grpcOpenTelemetry OTLP OTel gRPC | Input, Output | Certified | Yes | Yes | | otlp_httpOpenTelemetry OTLP OTel | Input, Output | Certified | Yes | Yes | | parallel | Processor | Certified | No | Yes | | parquet | Input, Processor | Certified: inputCommunity: processor | No | No | | parquet_decode | Processor | Certified | No | Yes | | parquet_encode | Processor | Certified | No | Yes | | parse_log | Processor | Community | No | Yes | | pg_stream | | Certified | Yes | Yes | | pinecone | Output | Certified | No | Yes | | postgres_cdc | Input | Certified | Yes | Yes | | processors | Processor | Certified | No | Yes | | prometheus | Metric | Certified | No | Yes | | protobuf | Processor | Certified | No | No | | pulsarApache Pulsar | Input, Output | Community | No | No | | pusher | Output | Community | No | No | | qdrant | Output, Processor | Certified | No | Yes | | questdb | Output | Certified | No | Yes | | rate_limit | Processor | Certified | No | Yes | | re_match | Scanner | Certified | No | Yes | | read_until | Input | Certified | No | Yes | | redis | Cache, Processor, Rate_limit | Certified | No | Yes | | redis_hashRedis Hash Redis | Output | Certified | No | Yes | | redis_listRedis List Redis Lists Redis | Input, Output | Certified | No | Yes | | redis_pubsubRedis PubSub Redis Pub/Sub Redis | Input, Output | Certified | No | Yes | | redis_scanRedis | Input | Certified | No | Yes | | redis_scriptRedis Script | Processor | Certified | No | Yes | | redis_streamsRedis Streams Redis | Input, Output | Certified | No | Yes | | redpanda | Cache, Input, Output, Tracer | Certified | No | Yes | | redpanda_common | Input, Output | Certified | Yes | Yes | | redpanda_data_transform | Processor | Certified | No | No | | redpanda_migrator | Input, Output | Certified | No | Yes | | reject | Output | Certified | No | Yes | | reject_errored | Output | Certified | No | Yes | | resource | Input, Output, Processor | Certified | No | Yes | | retry | Output, Processor | Certified | No | Yes | | ristretto | Cache | Community | No | Yes | | salesforce | Processor | Certified | Yes | No | | salesforce_sinkSalesforce Salesforce Sink | Output | Certified | Yes | No | | schema_registry | Input, Output | Certified | No | Yes | | schema_registry_decode | Processor | Certified | No | Yes | | schema_registry_encode | Processor | Certified | No | Yes | | select_parts | Processor | Certified | No | Yes | | sentry_capture | Processor | Community | No | No | | sequence | Input | Certified | No | Yes | | sftp | Input, Output | Certified | No | Yes | | skip_bom | Scanner | Certified | No | Yes | | slack | Input | Certified | Yes | Yes | | slack_postSlack Post | Output | Certified | Yes | Yes | | slack_reactionSlack Reaction | Output | Certified | Yes | Yes | | slack_threadSlack Thread | Processor | Certified | Yes | Yes | | slack_usersSlack Users | Input | Certified | Yes | Yes | | sleep | Processor | Certified | No | Yes | | snowflake_putSnowflake | Output | Certified | Yes | Yes | | snowflake_streamingSnowflake Streaming | Output | Certified | Yes | Yes | | socket | Input, Output | Certified | No | No | | socket_server | Input | Certified | No | No | | spicedb_watch | Input | Community | No | Yes | | split | Processor | Certified | No | Yes | | splunk | Input | Certified | Yes | Yes | | splunk_hecSplunk | Output | Certified | Yes | Yes | | sql | Cache, Output, Processor | Certified: cacheCommunity: output, processor | No | Yes | | sql_driver_clickhouseClickHouse | | Certified: MYSQL, Oracle, PostgreSQL, SQLiteCommunity: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino | No | Yes | | sql_driver_gocosmosAzure Cosmos DB | | Certified: MYSQL, Oracle, PostgreSQL, SQLiteCommunity: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino | No | No | | sql_driver_mssqlMicrosoft SQL Server | | Certified: MYSQL, Oracle, PostgreSQL, SQLiteCommunity: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino | No | No | | sql_driver_mysqlMYSQL | | Certified: MYSQL, Oracle, PostgreSQL, SQLiteCommunity: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino | No | Yes | | sql_driver_oracleOracle | | Certified: MYSQL, Oracle, PostgreSQL, SQLiteCommunity: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino | No | Yes | | sql_driver_postgresPostgreSQL | | Certified: MYSQL, Oracle, PostgreSQL, SQLiteCommunity: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino | No | Yes | | sql_driver_snowflakeSnowflake | | Certified: MYSQL, Oracle, PostgreSQL, SQLiteCommunity: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino | No | No | | sql_driver_sqliteSQLite | | Certified: MYSQL, Oracle, PostgreSQL, SQLiteCommunity: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino | No | Yes | | sql_driver_trinoTrino | | Certified: MYSQL, Oracle, PostgreSQL, SQLiteCommunity: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino | No | No | | sql_insertSQL PostgreSQL MySQL Microsoft SQL Server ClickHouse Trino | Output, Processor | Certified: MYSQL, Oracle, PostgreSQL, SQLiteCommunity: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino | No | Yes | | sql_rawSQL PostgreSQL MySQL Microsoft SQL Server ClickHouse Trino | Input, Output, Processor | Certified: MYSQL, Oracle, PostgreSQL, SQLiteCommunity: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino | No | Yes | | sql_selectSQL PostgreSQL MySQL Microsoft SQL Server ClickHouse Trino | Input, Processor | Certified: MYSQL, Oracle, PostgreSQL, SQLiteCommunity: ClickHouse, Azure Cosmos DB, Microsoft SQL Server, Snowflake, Trino | No | Yes | | sqlite | Buffer | Community | No | No | | statsd | Metric | Certified | No | No | | stdin | Input | Certified | No | No | | stdout | Output | Certified | No | No | | string_split | Processor | Certified | No | Yes | | subprocess | Input, Output, Processor | Community | No | No | | switch | Output, Processor, Scanner | Certified | No | Yes | | sync_response | Output, Processor | Certified | No | Yes | | system_window | Buffer | Certified | No | Yes | | tar | Scanner | Certified | No | Yes | | text_chunker | Processor | Certified | No | Yes | | tigerbeetle_cdc | Input | Certified | No | No | | timeplus | Input, Output | Community | No | Yes | | to_the_end | Scanner | Certified | No | Yes | | try | Processor | Certified | No | Yes | | ttlru | Cache | Community | No | Yes | | twitter_search | Input | Community | No | No | | unarchiveZIP TAR GZIP Archive | Processor | Certified | No | Yes | | wasm | Processor | Community | No | No | | websocket | Input, Output | Certified | No | No | | while | Processor | Certified | No | Yes | | workflow | Processor | Certified | No | Yes | | xml | Processor | Community | No | Yes | | zmq4 | Input, Output | Community | No | No | ## [](#about-components)About Components Every Redpanda Connect pipeline has at least one [input](../inputs/about/), an optional [buffer](../buffers/about/), an [output](../outputs/about/) and any number of [processors](../processors/about/): ```yaml input: kafka: addresses: [ TODO ] topics: [ foo, bar ] consumer_group: foogroup buffer: type: none pipeline: processors: - mapping: | message = this meta.link_count = links.length() output: aws_s3: bucket: TODO path: '${! meta("kafka_topic") }/${! json("message.id") }.json' ``` These are the main components within Redpanda Connect and they provide the majority of useful behavior. ## [](#observability-components)Observability components There are also the observability components: [http](../http/about/), [logger](../logger/about/), [metrics](../metrics/about/), and [tracing](../tracers/about/), which allow you to specify how Redpanda Connect exposes observability data. ```yaml http: address: 0.0.0.0:4195 enabled: true debug_endpoints: false logger: format: json level: WARN metrics: statsd: address: localhost:8125 flush_period: 100ms tracer: jaeger: agent_address: localhost:6831 ``` ## [](#resource-components)Resource components Finally, there are [caches](../caches/about/) and [rate limits](../rate_limits/about/). These are components that are referenced by core components and can be shared. ```yaml input: http_client: # This is an input url: TODO rate_limit: foo_ratelimit # This is a reference to a rate limit pipeline: processors: - cache: # This is a processor resource: baz_cache # This is a reference to a cache operator: add key: '${! json("id") }' value: "x" - mapping: root = if errored() { deleted() } rate_limit_resources: - label: foo_ratelimit local: count: 500 interval: 1s cache_resources: - label: baz_cache memcached: addresses: [ localhost:11211 ] ``` It’s also possible to configure inputs, outputs and processors as resources which allows them to be reused throughout a configuration with the [`resource` input](../inputs/resource/), [`resource` output](../outputs/resource/) and [`resource` processor](../processors/resource/) respectively. For more information about any of these component types check out their sections: - [inputs](../inputs/about/) - [processors](../processors/about/) - [outputs](../outputs/about/) - [buffers](../buffers/about/) - [metrics](../metrics/about/) - [tracers](../tracers/about/) - [logger](../logger/about/) - [caches](../caches/about/) - [rate limits](../rate_limits/about/) ## [](#support-levels)Support levels Each connector has a corresponding level of support, which determines whether it is suitable for production environments or community-driven projects. - **Certified**: Production-ready and available to purchase with Redpanda support. - **Community**: Not considered production-ready and available only for community support in [Slack](https://redpanda.com/slack). Anyone can use the Certified and Community connectors. However, to use Enterprise connectors, you need an [Enterprise Edition license](../../get-started/licensing/). --- # Page 10: Buffers **URL**: https://docs.redpanda.com/redpanda-connect/components/buffers/about.md --- # Buffers --- title: Buffers latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: buffers/about page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: buffers/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/buffers/about.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- Redpanda Connect uses a transaction model internally for guaranteeing delivery of messages, this means that a message from an input is not acknowledged (or its offset committed, etc) until that message has been processed and either intentionally deleted or successfully delivered to all outputs. This transaction model makes Redpanda Connect safe to deploy in scenarios where data loss is unacceptable. However, sometimes it’s useful to customize the way in which messages are delivered, and this is where buffers come in. A buffer is an optional component type that comes immediately after the input layer and can be used as a way of decoupling the transaction model from components downstream such as the processing layer and outputs. This is considered an advanced component as most users will likely not benefit from a buffer, but they enable you to do things like group messages using window algorithms or intentionally weaken the delivery guarantees of the pipeline depending on the buffer you choose. Since buffers are able to modify (or disable) the transaction model within Redpanda Connect it is important that when you choose a buffer you read its documentation to understand the implication it will have on delivery guarantees. ## [](#categories)Categories ### Utility Buffers that are intended for niche but general use. [**memory**](/redpanda-connect/components/buffers/memory/) [**sqlite**](/redpanda-connect/components/buffers/sqlite/) ### Windowing Buffers that provide message windowing capabilities. [**system\_window**](/redpanda-connect/components/buffers/system_window/) --- # Page 11: memory **URL**: https://docs.redpanda.com/redpanda-connect/components/buffers/memory.md --- # memory --- title: memory latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: buffers/memory page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: buffers/memory.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/buffers/memory.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Buffer ▼ [Buffer](/redpanda-connect/components/buffers/memory/)[Cache](/redpanda-connect/components/caches/memory/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/buffers/memory/ "View the Cloud version of this component"), Self-Managed Stores consumed messages in memory and acknowledges them at the input level. During shutdown Redpanda Connect will make a best attempt at flushing all remaining messages before exiting cleanly. #### Common ```yml buffers: memory: limit: 524288000 batch_policy: enabled: false count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml buffers: memory: limit: 524288000 batch_policy: enabled: false count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` This buffer is appropriate when consuming messages from inputs that do not gracefully handle back pressure and where delivery guarantees aren’t critical. This buffer has a configurable limit, where consumption will be stopped with back pressure upstream if the total size of messages in the buffer reaches this amount. Since this calculation is only an estimate, and the real size of messages in RAM is always higher, it is recommended to set the limit significantly below the amount of RAM available. ## [](#delivery-guarantees)Delivery guarantees This buffer intentionally weakens the delivery guarantees of the pipeline and therefore should never be used in places where data loss is unacceptable. ## [](#batching)Batching It is possible to batch up messages sent from this buffer using a [batch policy](../../../configuration/batching/#batch-policy). ## [](#fields)Fields ### [](#batch_policy)`batch_policy` Optionally configure a policy to flush buffered messages in batches. **Type**: `object` ### [](#batch_policy-byte_size)`batch_policy.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batch_policy-check)`batch_policy.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batch_policy-count)`batch_policy.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batch_policy-enabled)`batch_policy.enabled` Whether to batch messages as they are flushed. **Type**: `bool` **Default**: `false` ### [](#batch_policy-period)`batch_policy.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batch_policy-processors)`batch_policy.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#limit)`limit` The maximum buffer size (in bytes) to allow before applying backpressure upstream. **Type**: `int` **Default**: `524288000` --- # Page 12: none **URL**: https://docs.redpanda.com/redpanda-connect/components/buffers/none.md --- # none --- title: none latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: buffers/none page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: buffers/none.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/buffers/none.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- **Type:** Buffer ▼ [Buffer](/redpanda-connect/components/buffers/none/)[Metric](/redpanda-connect/components/metrics/none/)[Tracer](/redpanda-connect/components/tracers/none/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/buffers/none/ "View the Cloud version of this component"), Self-Managed Do not buffer messages. This is the default and most resilient configuration. ```yml # Config fields, showing default values buffer: none: {} ``` Selecting no buffer means the output layer is directly coupled with the input layer. This is the safest and lowest latency option since acknowledgements from at-least-once protocols can be propagated all the way from the output protocol to the input protocol. If the output layer is hit with back pressure it will propagate all the way to the input layer, and further up the data stream. If you need to relieve your pipeline of this back pressure consider using a more robust buffering solution such as Kafka before resorting to alternatives. --- # Page 13: sqlite **URL**: https://docs.redpanda.com/redpanda-connect/components/buffers/sqlite.md --- # sqlite --- title: sqlite latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: buffers/sqlite page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: buffers/sqlite.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/buffers/sqlite.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Available in:** Self-Managed Stores messages in an SQLite database and acknowledges them at the input level. #### Common ```yml buffers: sqlite: path: "" # No default (required) pre_processors: [] # No default (optional) post_processors: [] # No default (optional) ``` #### Advanced ```yml buffers: sqlite: path: "" # No default (required) pre_processors: [] # No default (optional) post_processors: [] # No default (optional) ``` Stored messages are then consumed as a stream from the database and deleted only once they are successfully sent at the output level. If the service is restarted Redpanda Connect will make a best attempt to finish delivering messages that are already read from the database, and when it starts again it will consume from the oldest message that has not yet been delivered. ## [](#delivery-guarantees)Delivery guarantees Messages are not acknowledged at the input level until they have been added to the SQLite database, and they are not removed from the SQLite database until they have been successfully delivered. This means at-least-once delivery guarantees are preserved in cases where the service is shut down unexpectedly. However, since this process relies on interaction with the disk (wherever the SQLite DB is stored) these delivery guarantees are not resilient to disk corruption or loss. ## [](#batching)Batching Messages that are logically batched at the point where they are added to the buffer will continue to be associated with that batch when they are consumed. This buffer is also more efficient when storing messages within batches, and therefore it is recommended to use batching at the input level in high-throughput use cases even if they are not required for processing. ## [](#fields)Fields ### [](#path)`path` The path of the database file, which will be created if it does not already exist. **Type**: `string` ### [](#post_processors)`post_processors[]` An optional list of processors to apply to messages after they are consumed from the buffer. These processors are useful for undoing any compression, archiving, etc that may have been done by your `pre_processors`. **Type**: `processor` ### [](#pre_processors)`pre_processors[]` An optional list of processors to apply to messages before they are stored within the buffer. These processors are useful for compressing, archiving or otherwise reducing the data in size before it’s stored on disk. **Type**: `processor` nclude::redpanda-connect:components:partial$examples/buffers/sqlite.adoc\[\] --- # Page 14: system_window **URL**: https://docs.redpanda.com/redpanda-connect/components/buffers/system_window.md --- # system\_window --- title: system_window latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: buffers/system_window page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: buffers/system_window.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/buffers/system_window.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/buffers/system_window/ "View the Cloud version of this component"), Self-Managed Chops a stream of messages into tumbling or sliding windows of fixed temporal size, following the system clock. Introduced in version 3.53.0. #### Common ```yml buffers: system_window: timestamp_mapping: root = now() size: "" # No default (required) slide: "" offset: "" allowed_lateness: "" ``` #### Advanced ```yml buffers: system_window: timestamp_mapping: root = now() size: "" # No default (required) slide: "" offset: "" allowed_lateness: "" ``` A window is a grouping of messages that fit within a discrete measure of time following the system clock. Messages are allocated to a window either by the processing time (the time at which they’re ingested) or by the event time, and this is controlled via the [`timestamp_mapping` field](#timestamp_mapping). In tumbling mode (default) the beginning of a window immediately follows the end of a prior window. When the buffer is initialized the first window to be created and populated is aligned against the zeroth minute of the zeroth hour of the day by default, and may therefore be open for a shorter period than the specified size. A window is flushed only once the system clock surpasses its scheduled end. If an [`allowed_lateness`](#allowed_lateness) is specified then the window will not be flushed until the scheduled end plus that length of time. When a message is added to a window it has a metadata field `window_end_timestamp` added to it containing the timestamp of the end of the window as an RFC3339 string. ## [](#sliding-windows)Sliding windows Sliding windows begin from an offset of the prior windows' beginning rather than its end, and therefore messages may belong to multiple windows. In order to produce sliding windows specify a [`slide` duration](#slide). ## [](#back-pressure)Back pressure If back pressure is applied to this buffer either due to output services being unavailable or resources being saturated, windows older than the current and last according to the system clock will be dropped in order to prevent unbounded resource usage. This means you should ensure that under the worst case scenario you have enough system memory to store two windows' worth of data at a given time (plus extra for redundancy and other services). If messages could potentially arrive with event timestamps in the future (according to the system clock) then you should also factor in these extra messages in memory usage estimates. ## [](#delivery-guarantees)Delivery guarantees This buffer honours the transaction model within Redpanda Connect in order to ensure that messages are not acknowledged until they are either intentionally dropped or successfully delivered to outputs. However, since messages belonging to an expired window are intentionally dropped there are circumstances where not all messages entering the system will be delivered. When this buffer is configured with a slide duration it is possible for messages to belong to multiple windows, and therefore be delivered multiple times. In this case the first time the message is delivered it will be acked (or nacked) and subsequent deliveries of the same message will be a "best attempt". During graceful termination if the current window is partially populated with messages they will be nacked such that they are re-consumed the next time the service starts. ## [](#examples)Examples ### Counting Passengers at Traffic Given a stream of messages relating to cars passing through various traffic lights of the form: ```json { "traffic_light": "cbf2eafc-806e-4067-9211-97be7e42cee3", "created_at": "2021-08-07T09:49:35Z", "registration_plate": "AB1C DEF", "passengers": 3 } ``` We can use a window buffer in order to create periodic messages summarizing the traffic for a period of time of this form: ```json { "traffic_light": "cbf2eafc-806e-4067-9211-97be7e42cee3", "created_at": "2021-08-07T10:00:00Z", "total_cars": 15, "passengers": 43 } ``` With the following config: ```yaml buffer: system_window: timestamp_mapping: root = this.created_at size: 1h pipeline: processors: # Group messages of the window into batches of common traffic light IDs - group_by_value: value: '${! json("traffic_light") }' # Reduce each batch to a single message by deleting indexes > 0, and # aggregate the car and passenger counts. - mapping: | root = if batch_index() == 0 { { "traffic_light": this.traffic_light, "created_at": meta("window_end_timestamp"), "total_cars": json("registration_plate").from_all().unique().length(), "passengers": json("passengers").from_all().sum(), } } else { deleted() } ``` ## [](#fields)Fields ### [](#timestamp_mapping)`timestamp_mapping` A [Bloblang mapping](../../../guides/bloblang/about/) applied to each message during ingestion that provides the timestamp to use for allocating it a window. By default the function `now()` is used in order to generate a fresh timestamp at the time of ingestion (the processing time), whereas this mapping can instead extract a timestamp from the message itself (the event time). The timestamp value assigned to `root` must either be a numerical unix time in seconds (with up to nanosecond precision via decimals), or a string in ISO 8601 format. If the mapping fails or provides an invalid result the message will be dropped (with logging to describe the problem). **Type**: `string` **Default**: `"root = now()"` ```yml # Examples timestamp_mapping: root = this.created_at timestamp_mapping: root = meta("kafka_timestamp_unix").number() ``` ### [](#size)`size` A duration string describing the size of each window. By default windows are aligned to the zeroth minute and zeroth hour on the UTC clock, meaning windows of 1 hour duration will match the turn of each hour in the day, this can be adjusted with the `offset` field. **Type**: `string` ```yml # Examples size: 30s size: 10m ``` ### [](#slide)`slide` An optional duration string describing by how much time the beginning of each window should be offset from the beginning of the previous, and therefore creates sliding windows instead of tumbling. When specified this duration must be smaller than the `size` of the window. **Type**: `string` **Default**: `""` ```yml # Examples slide: 30s slide: 10m ``` ### [](#offset)`offset` An optional duration string to offset the beginning of each window by, otherwise they are aligned to the zeroth minute and zeroth hour on the UTC clock. The offset cannot be a larger or equal measure to the window size or the slide. **Type**: `string` **Default**: `""` ```yml # Examples offset: -6h offset: 30m ``` ### [](#allowed_lateness)`allowed_lateness` An optional duration string describing the length of time to wait after a window has ended before flushing it, allowing late arrivals to be included. Since this windowing buffer uses the system clock an allowed lateness can improve the matching of messages when using event time. **Type**: `string` **Default**: `""` ```yml # Examples allowed_lateness: 10s allowed_lateness: 1m ``` --- # Page 15: Caches **URL**: https://docs.redpanda.com/redpanda-connect/components/caches/about.md --- # Caches --- title: Caches latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: caches/about page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/about.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-09" --- A cache is a key/value store which can be used by certain components for applications such as deduplication or data joins. Caches are configured as a named resource: ```yaml cache_resources: - label: foobar memcached: addresses: - localhost:11211 default_ttl: 60s ``` > It’s possible to layer caches with read-through and write-through behavior using the [`multilevel` cache](../multilevel/). And then any components that use caches have a field `resource` that specifies the cache resource: ```yaml pipeline: processors: - cache: resource: foobar operator: add key: '${! json("message.id") }' value: "storeme" - mapping: root = if errored() { deleted() } ``` For the simple case where you wish to store messages in a cache as an output destination for your pipeline check out the [`cache` output](../../outputs/cache/). To see examples of more advanced uses of caches such as hydration and deduplication check out the [`cache` processor](../../processors/cache/). You can find out more about resources [in this document.](../../../configuration/resources/) --- # Page 16: aws_dynamodb **URL**: https://docs.redpanda.com/redpanda-connect/components/caches/aws_dynamodb.md --- # aws\_dynamodb --- title: aws_dynamodb latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: caches/aws_dynamodb page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/aws_dynamodb.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/aws_dynamodb.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Cache ▼ [Cache](/redpanda-connect/components/caches/aws_dynamodb/)[Output](/redpanda-connect/components/outputs/aws_dynamodb/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/caches/aws_dynamodb/ "View the Cloud version of this component"), Self-Managed Stores key/value pairs as a single document in a DynamoDB table. The key is stored as a string value and used as the table hash key. The value is stored as a binary value using the `data_key` field name. Introduced in version 3.36.0. #### Common ```yml caches: aws_dynamodb: table: "" # No default (required) hash_key: "" # No default (required) data_key: "" # No default (required) ``` #### Advanced ```yml caches: aws_dynamodb: table: "" # No default (required) hash_key: "" # No default (required) data_key: "" # No default (required) consistent_read: false default_ttl: "" # No default (optional) ttl_key: "" # No default (optional) retries: initial_interval: 1s max_interval: 5s max_elapsed_time: 30s region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` A prefix can be specified to allow multiple cache types to share a single DynamoDB table. An optional TTL duration (`ttl`) and field (`ttl_key`) can be specified if the backing table has TTL enabled. Strong read consistency can be enabled using the `consistent_read` configuration field. ## [](#fields)Fields ### [](#consistent_read)`consistent_read` Whether to use strongly consistent reads on Get commands. **Type**: `bool` **Default**: `false` ### [](#credentials)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](../../../guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#data_key)`data_key` The key of the table column to store item values within. **Type**: `string` ### [](#default_ttl)`default_ttl` An optional default TTL to set for items, calculated from the moment the item is cached. A `ttl_key` must be specified in order to set item TTLs. **Type**: `string` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#hash_key)`hash_key` The key of the table column to store item keys within. **Type**: `string` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#retries)`retries` Determine time intervals and cut offs for retry attempts. **Type**: `object` ### [](#retries-initial_interval)`retries.initial_interval` The initial period to wait between retry attempts. **Type**: `string` **Default**: `1s` ```yaml # Examples: initial_interval: 50ms # --- initial_interval: 1s ``` ### [](#retries-max_elapsed_time)`retries.max_elapsed_time` The maximum overall period of time to spend on retry attempts before the request is aborted. **Type**: `string` **Default**: `30s` ```yaml # Examples: max_elapsed_time: 1m # --- max_elapsed_time: 1h ``` ### [](#retries-max_interval)`retries.max_interval` The maximum period to wait between retry attempts **Type**: `string` **Default**: `5s` ```yaml # Examples: max_interval: 5s # --- max_interval: 1m ``` ### [](#table)`table` The table to store items in. **Type**: `string` ### [](#tcp)`tcp` TCP socket configuration. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#ttl_key)`ttl_key` The column key to place the TTL value within. **Type**: `string` --- # Page 17: aws_s3 **URL**: https://docs.redpanda.com/redpanda-connect/components/caches/aws_s3.md --- # aws\_s3 --- title: aws_s3 latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: caches/aws_s3 page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/aws_s3.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/aws_s3.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Cache ▼ [Cache](/redpanda-connect/components/caches/aws_s3/)[Input](/redpanda-connect/components/inputs/aws_s3/)[Output](/redpanda-connect/components/outputs/aws_s3/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/caches/aws_s3/ "View the Cloud version of this component"), Self-Managed Stores each item in an S3 bucket as a file, where an item ID is the path of the item within the bucket. Introduced in version 3.36.0. #### Common ```yml caches: aws_s3: bucket: "" # No default (required) content_type: application/octet-stream ``` #### Advanced ```yml caches: aws_s3: bucket: "" # No default (required) content_type: application/octet-stream force_path_style_urls: false retries: initial_interval: 1s max_interval: 5s max_elapsed_time: 30s region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` It is not possible to atomically upload S3 objects exclusively when the target does not already exist, therefore this cache is not suitable for deduplication. ## [](#fields)Fields ### [](#bucket)`bucket` The S3 bucket to store items in. **Type**: `string` ### [](#content_type)`content_type` The content type to set for each item. **Type**: `string` **Default**: `application/octet-stream` ### [](#credentials)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](../../../guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#force_path_style_urls)`force_path_style_urls` Forces the client API to use path style URLs, which helps when connecting to custom endpoints. **Type**: `bool` **Default**: `false` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#retries)`retries` Determine time intervals and cut offs for retry attempts. **Type**: `object` ### [](#retries-initial_interval)`retries.initial_interval` The initial period to wait between retry attempts. **Type**: `string` **Default**: `1s` ```yaml # Examples: initial_interval: 50ms # --- initial_interval: 1s ``` ### [](#retries-max_elapsed_time)`retries.max_elapsed_time` The maximum overall period of time to spend on retry attempts before the request is aborted. **Type**: `string` **Default**: `30s` ```yaml # Examples: max_elapsed_time: 1m # --- max_elapsed_time: 1h ``` ### [](#retries-max_interval)`retries.max_interval` The maximum period to wait between retry attempts **Type**: `string` **Default**: `5s` ```yaml # Examples: max_interval: 5s # --- max_interval: 1m ``` ### [](#tcp)`tcp` TCP socket configuration. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` --- # Page 18: couchbase **URL**: https://docs.redpanda.com/redpanda-connect/components/caches/couchbase.md --- # couchbase --- title: couchbase latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: caches/couchbase page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/couchbase.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/couchbase.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Cache ▼ [Cache](/redpanda-connect/components/caches/couchbase/)[Output](/redpanda-connect/components/outputs/couchbase/)[Processor](/redpanda-connect/components/processors/couchbase/) **Available in:** Self-Managed Use a Couchbase instance as a cache. Introduced in version 4.12.0. #### Common ```yml caches: couchbase: url: "" # No default (required) username: "" # No default (optional) password: "" # No default (optional) bucket: "" # No default (required) ``` #### Advanced ```yml caches: couchbase: url: "" # No default (required) username: "" # No default (optional) password: "" # No default (optional) bucket: "" # No default (required) collection: "" # No default (optional) scope: "" # No default (optional) transcoder: legacy timeout: 15s default_ttl: "" # No default (optional) ``` ## [](#fields)Fields ### [](#bucket)`bucket` Couchbase bucket. **Type**: `string` ### [](#collection)`collection` Bucket collection. **Type**: `string` ### [](#default_ttl)`default_ttl` An optional default TTL to set for items, calculated from the moment the item is cached. **Type**: `string` ### [](#password)`password` Password to connect to the cluster. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#scope)`scope` Bucket scope. **Type**: `string` ### [](#timeout)`timeout` Operation timeout. **Type**: `string` **Default**: `15s` ### [](#transcoder)`transcoder` Couchbase transcoder to use. **Type**: `string` **Default**: `legacy` | Option | Summary | | --- | --- | | json | JSONTranscoder implements the default transcoding behavior and applies JSON transcoding to all values. This will apply the following behavior to the value: binary ([]byte) → error. default → JSON value, JSON Flags. | | legacy | LegacyTranscoder implements the behavior for a backward-compatible transcoder. This transcoder implements behavior matching that of gocb v1.This will apply the following behavior to the value: binary ([]byte) → binary bytes, Binary expectedFlags. string → string bytes, String expectedFlags. default → JSON value, JSON expectedFlags. | | raw | RawBinaryTranscoder implements passthrough behavior of raw binary data. This transcoder does not apply any serialization. This will apply the following behavior to the value: binary ([]byte) → binary bytes, binary expectedFlags. default → error. | | rawjson | RawJSONTranscoder implements passthrough behavior of JSON data. This transcoder does not apply any serialization. It will forward data across the network without incurring unnecessary parsing costs. This will apply the following behavior to the value: binary ([]byte) → JSON bytes, JSON expectedFlags. string → JSON bytes, JSON expectedFlags. default → error. | | rawstring | RawStringTranscoder implements passthrough behavior of raw string data. This transcoder does not apply any serialization. This will apply the following behavior to the value: string → string bytes, string expectedFlags. default → error. | ### [](#url)`url` Couchbase connection string. **Type**: `string` ```yaml # Examples: url: couchbase://localhost:11210 ``` ### [](#username)`username` Username to connect to the cluster. **Type**: `string` --- # Page 19: file **URL**: https://docs.redpanda.com/redpanda-connect/components/caches/file.md --- # file --- title: file latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: caches/file page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/file.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/file.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Cache ▼ [Cache](/redpanda-connect/components/caches/file/)[Input](/redpanda-connect/components/inputs/file/)[Output](/redpanda-connect/components/outputs/file/) **Available in:** Self-Managed Stores each item in a directory as a file, where an item ID is the path relative to the configured directory. ```yml caches: file: directory: "" # No default (required) ``` This type currently offers no form of item expiry or garbage collection, and is intended to be used for development and debugging purposes only. ## [](#fields)Fields ### [](#directory)`directory` The directory within which to store items. **Type**: `string` --- # Page 20: gcp_cloud_storage **URL**: https://docs.redpanda.com/redpanda-connect/components/caches/gcp_cloud_storage.md --- # gcp\_cloud\_storage --- title: gcp_cloud_storage latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: caches/gcp_cloud_storage page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/gcp_cloud_storage.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/gcp_cloud_storage.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Cache ▼ [Cache](/redpanda-connect/components/caches/gcp_cloud_storage/)[Input](/redpanda-connect/components/inputs/gcp_cloud_storage/)[Output](/redpanda-connect/components/outputs/gcp_cloud_storage/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/caches/gcp_cloud_storage/ "View the Cloud version of this component"), Self-Managed Use a Google Cloud Storage bucket as a cache. ```yml caches: gcp_cloud_storage: bucket: "" # No default (required) content_type: "" # No default (optional) credentials_json: "" ``` It is not possible to atomically upload cloud storage objects exclusively when the target does not already exist, therefore this cache is not suitable for deduplication. ## [](#fields)Fields ### [](#bucket)`bucket` The Google Cloud Storage bucket to store items in. **Type**: `string` ### [](#content_type)`content_type` Optional field to explicitly set the Content-Type. **Type**: `string` ### [](#credentials_json)`credentials_json` An optional field to set Google Service Account Credentials json. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` --- # Page 21: lru **URL**: https://docs.redpanda.com/redpanda-connect/components/caches/lru.md --- # lru --- title: lru latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: caches/lru page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/lru.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/lru.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/caches/lru/ "View the Cloud version of this component"), Self-Managed Stores key/value pairs in a lru in-memory cache. This cache is therefore reset every time the service restarts. #### Common ```yml caches: lru: cap: 1000 init_values: {} ``` #### Advanced ```yml caches: lru: cap: 1000 init_values: {} algorithm: standard two_queues_recent_ratio: 0.25 two_queues_ghost_ratio: 0.5 optimistic: false ``` This provides the lru package which implements a fixed-size thread safe LRU cache. It uses the package [`lru`](https://github.com/hashicorp/golang-lru/v2) The field init\_values can be used to pre-populate the memory cache with any number of key/value pairs: ```yaml cache_resources: - label: foocache lru: cap: 1024 init_values: foo: bar ``` These values can be overridden during execution. ## [](#fields)Fields ### [](#algorithm)`algorithm` the lru cache implementation **Type**: `string` **Default**: `standard` | Option | Summary | | --- | --- | | arc | is an adaptive replacement cache. It tracks recent evictions as well as recent usage in both the frequent and recent caches. Its computational overhead is comparable to two_queues, but the memory overhead is linear with the size of the cache. ARC has been patented by IBM. | | standard | is a simple LRU cache. It is based on the LRU implementation in groupcache | | two_queues | tracks frequently used and recently used entries separately. This avoids a burst of accesses from taking out frequently used entries, at the cost of about 2x computational overhead and some extra bookkeeping. | ### [](#cap)`cap` The cache maximum capacity (number of entries) **Type**: `int` **Default**: `1000` ### [](#init_values)`init_values` A table of key/value pairs that should be present in the cache on initialization. This can be used to create static lookup tables. **Type**: `string` **Default**: `{}` ```yaml # Examples: init_values: Nickelback: "1995" Spice Girls: "1994" The Human League: "1977" ``` ### [](#optimistic)`optimistic` If true, we do not lock on read/write events. The lru package is thread-safe, however the ADD operation is not atomic. **Type**: `bool` **Default**: `false` ### [](#two_queues_ghost_ratio)`two_queues_ghost_ratio` is the default ratio of ghost entries kept to track entries recently evicted on two\_queues cache. **Type**: `float` **Default**: `0.5` ### [](#two_queues_recent_ratio)`two_queues_recent_ratio` is the ratio of the two\_queues cache dedicated to recently added entries that have only been accessed once. **Type**: `float` **Default**: `0.25` --- # Page 22: memcached **URL**: https://docs.redpanda.com/redpanda-connect/components/caches/memcached.md --- # memcached --- title: memcached latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: caches/memcached page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/memcached.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/memcached.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/caches/memcached/ "View the Cloud version of this component"), Self-Managed Connects to a cluster of memcached services, a prefix can be specified to allow multiple cache types to share a memcached cluster under different namespaces. #### Common ```yml caches: memcached: addresses: [] # No default (required) prefix: "" # No default (optional) default_ttl: 300s ``` #### Advanced ```yml caches: memcached: addresses: [] # No default (required) prefix: "" # No default (optional) default_ttl: 300s retries: initial_interval: 1s max_interval: 5s max_elapsed_time: 30s ``` ## [](#fields)Fields ### [](#addresses)`addresses[]` A list of addresses of memcached servers to use. **Type**: `array` ### [](#default_ttl)`default_ttl` A default TTL to set for items, calculated from the moment the item is cached. **Type**: `string` **Default**: `300s` ### [](#prefix)`prefix` An optional string to prefix item keys with in order to prevent collisions with similar services. **Type**: `string` ### [](#retries)`retries` Determine time intervals and cut offs for retry attempts. **Type**: `object` ### [](#retries-initial_interval)`retries.initial_interval` The initial period to wait between retry attempts. **Type**: `string` **Default**: `1s` ```yaml # Examples: initial_interval: 50ms # --- initial_interval: 1s ``` ### [](#retries-max_elapsed_time)`retries.max_elapsed_time` The maximum overall period of time to spend on retry attempts before the request is aborted. **Type**: `string` **Default**: `30s` ```yaml # Examples: max_elapsed_time: 1m # --- max_elapsed_time: 1h ``` ### [](#retries-max_interval)`retries.max_interval` The maximum period to wait between retry attempts **Type**: `string` **Default**: `5s` ```yaml # Examples: max_interval: 5s # --- max_interval: 1m ``` --- # Page 23: memory **URL**: https://docs.redpanda.com/redpanda-connect/components/caches/memory.md --- # memory --- title: memory latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: caches/memory page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/memory.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/memory.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Cache ▼ [Cache](/redpanda-connect/components/caches/memory/)[Buffer](/redpanda-connect/components/buffers/memory/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/caches/memory/ "View the Cloud version of this component"), Self-Managed Stores key/value pairs in a map held in memory. This cache is therefore reset every time the service restarts. Each item in the cache has a TTL set from the moment it was last edited, after which it will be removed during the next compaction. #### Common ```yml caches: memory: default_ttl: 5m compaction_interval: 60s init_values: {} ``` #### Advanced ```yml caches: memory: default_ttl: 5m compaction_interval: 60s init_values: {} shards: 1 ``` The compaction interval determines how often the cache is cleared of expired items, and this process is only triggered on writes to the cache. Access to the cache is blocked during this process. Item expiry can be disabled entirely by setting the `compaction_interval` to an empty string. The field `init_values` can be used to prepopulate the memory cache with any number of key/value pairs which are exempt from TTLs: ```yaml cache_resources: - label: foocache memory: default_ttl: 60s init_values: foo: bar ``` These values can be overridden during execution, at which point the configured TTL is respected as usual. ## [](#fields)Fields ### [](#compaction_interval)`compaction_interval` The period of time to wait before each compaction, at which point expired items are removed. This field can be set to an empty string in order to disable compactions/expiry entirely. **Type**: `string` **Default**: `60s` ### [](#default_ttl)`default_ttl` The default TTL of each item. After this period an item will be eligible for removal during the next compaction. **Type**: `string` **Default**: `5m` ### [](#init_values)`init_values` A table of key/value pairs that should be present in the cache on initialization. This can be used to create static lookup tables. **Type**: `string` **Default**: `{}` ```yaml # Examples: init_values: Nickelback: "1995" Spice Girls: "1994" The Human League: "1977" ``` ### [](#shards)`shards` A number of logical shards to spread keys across, increasing the shards can have a performance benefit when processing a large number of keys. **Type**: `int` **Default**: `1` --- # Page 24: mongodb **URL**: https://docs.redpanda.com/redpanda-connect/components/caches/mongodb.md --- # mongodb --- title: mongodb latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: caches/mongodb page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/mongodb.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/mongodb.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Cache ▼ [Cache](/redpanda-connect/components/caches/mongodb/)[Input](/redpanda-connect/components/inputs/mongodb/)[Output](/redpanda-connect/components/outputs/mongodb/)[Processor](/redpanda-connect/components/processors/mongodb/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/caches/mongodb/ "View the Cloud version of this component"), Self-Managed Use a MongoDB instance as a cache. Introduced in version 3.43.0. #### Common ```yml caches: mongodb: url: "" # No default (required) database: "" # No default (required) username: "" password: "" collection: "" # No default (required) key_field: "" # No default (required) value_field: "" # No default (required) ``` #### Advanced ```yml caches: mongodb: url: "" # No default (required) database: "" # No default (required) username: "" password: "" app_name: benthos collection: "" # No default (required) key_field: "" # No default (required) value_field: "" # No default (required) ``` ## [](#fields)Fields ### [](#app_name)`app_name` The client application name. **Type**: `string` **Default**: `benthos` ### [](#collection)`collection` The name of the target collection. **Type**: `string` ### [](#database)`database` The name of the target MongoDB database. **Type**: `string` ### [](#key_field)`key_field` The field in the document that is used as the key. **Type**: `string` ### [](#password)`password` The password to connect to the database. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#url)`url` The URL of the target MongoDB server. **Type**: `string` ```yaml # Examples: url: mongodb://localhost:27017 ``` ### [](#username)`username` The username to connect to the database. **Type**: `string` **Default**: `""` ### [](#value_field)`value_field` The field in the document that is used as the value. **Type**: `string` --- # Page 25: multilevel **URL**: https://docs.redpanda.com/redpanda-connect/components/caches/multilevel.md --- # multilevel --- title: multilevel latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: caches/multilevel page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/multilevel.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/multilevel.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/caches/multilevel/ "View the Cloud version of this component"), Self-Managed Combines multiple caches as levels, performing read-through and write-through operations across them. ```yml caches: multilevel: - label: "" memory: default_ttl: 5m compaction_interval: 60s - label: "" redis: url: redis://localhost:6379 expiration: 24h ``` ## [](#examples)Examples ### [](#hot-and-cold-cache)Hot and cold cache The multilevel cache is useful for reducing traffic against a remote cache by routing it through a local cache. In the following example requests will only go through to the memcached server if the local memory cache is missing the key. ```yaml pipeline: processors: - branch: processors: - cache: resource: leveled operator: get key: ${! json("key") } - catch: - mapping: 'root = {"err":error()}' result_map: 'root.result = this' cache_resources: - label: leveled multilevel: [ hot, cold ] - label: hot memory: default_ttl: 60s - label: cold memcached: addresses: [ TODO:11211 ] default_ttl: 60s ``` --- # Page 26: nats_kv **URL**: https://docs.redpanda.com/redpanda-connect/components/caches/nats_kv.md --- # nats\_kv --- title: nats_kv latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: caches/nats_kv page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/nats_kv.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/nats_kv.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Cache ▼ [Cache](/redpanda-connect/components/caches/nats_kv/)[Input](/redpanda-connect/components/inputs/nats_kv/)[Output](/redpanda-connect/components/outputs/nats_kv/)[Processor](/redpanda-connect/components/processors/nats_kv/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/caches/nats_kv/ "View the Cloud version of this component"), Self-Managed Cache key/value pairs in a NATS key-value bucket. Introduced in version 4.27.0. #### Common ```yml caches: nats_kv: urls: [] # No default (required) bucket: "" # No default (required) ``` #### Advanced ```yml caches: nats_kv: urls: [] # No default (required) max_reconnects: "" # No default (optional) bucket: "" # No default (required) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) ``` ## [](#connection-name)Connection name When monitoring and managing a production [NATS system](https://docs.nats.io/nats-concepts/overview), it is often useful to know which connection a message was sent or received from. To achieve this, set the connection name option when creating a NATS connection. Redpanda Connect can then automatically set the connection name to the NATS component label, so that monitoring tools between NATS and Redpanda Connect can stay in sync. ## [](#authentication)Authentication A number of Redpanda Connect components use NATS services. Each of these components support optional, advanced authentication parameters for [NKeys](https://docs.nats.io/nats-server/configuration/securing_nats/auth_intro/nkey_auth) and [user credentials](https://docs.nats.io/using-nats/developer/connecting/creds). For an in-depth guide, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/nats_admin/security/jwt). ### [](#nkeys)NKeys NATS server can use NKeys in several ways for authentication. The simplest approach is to configure the server with a list of user’s public keys. The server can then generate a challenge for each connection request from a client, and the client must respond to the challenge by signing it with its private NKey, configured in the `nkey_file` or `nkey` field. For more details, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth). ### [](#user-credentials)User credentials NATS server also supports decentralized authentication based on JSON Web Tokens (JWTs). When a server is configured to use this authentication scheme, clients need a [user JWT](https://docs.nats.io/nats-server/configuration/securing_nats/jwt#json-web-tokens) and a corresponding [NKey secret](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth) to connect. You can use either of the following methods to supply the user JWT and NKey secret: - In the `user_credentials_file` field, enter the path to a file containing both the private key and the JWT. You can generate the file using the [nsc tool](https://docs.nats.io/nats-tools/nsc). - In the `user_jwt` field, enter a plain text JWT, and in the `user_nkey_seed` field, enter the plain text NKey seed or private key. For more details about authentication using JWTs, see the [NATS documentation](https://docs.nats.io/using-nats/developer/connecting/creds). ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` The NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of an user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plain text user JWT (given along with the corresponding user NKey Seed). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plain text user NKey Seed (given along with the corresponding user JWT). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#bucket)`bucket` The name of the KV bucket. **Type**: `string` ```yaml # Examples: bucket: my_kv_bucket ``` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. **Type**: `int` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. Requires version 4.60.0 or later. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If an item of the list contains commas it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 27: noop **URL**: https://docs.redpanda.com/redpanda-connect/components/caches/noop.md --- # noop --- title: noop latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: caches/noop page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/noop.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/noop.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Cache ▼ [Cache](/redpanda-connect/components/caches/noop/)[Processor](/redpanda-connect/components/processors/noop/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/caches/noop/ "View the Cloud version of this component"), Self-Managed Noop is a cache that stores nothing, all gets returns not found. Why? Sometimes doing nothing is the braver option. Introduced in version 4.27.0. ```yml caches: noop: {} ``` --- # Page 28: redis **URL**: https://docs.redpanda.com/redpanda-connect/components/caches/redis.md --- # redis --- title: redis latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: caches/redis page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/redis.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/redis.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Cache ▼ [Cache](/redpanda-connect/components/caches/redis/)[Processor](/redpanda-connect/components/processors/redis/)[Rate\_limit](/redpanda-connect/components/rate_limits/redis/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/caches/redis/ "View the Cloud version of this component"), Self-Managed Use a Redis instance as a cache. The expiration can be set to zero or an empty string in order to set no expiration. #### Common ```yml caches: redis: url: "" # No default (required) prefix: "" # No default (optional) ``` #### Advanced ```yml caches: redis: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] prefix: "" # No default (optional) default_ttl: "" # No default (optional) retries: initial_interval: 500ms max_interval: 1s max_elapsed_time: 5s ``` ## [](#fields)Fields ### [](#client_name)`client_name` Set the client name for the Redis connection. Requires version 4.82.0 or later. **Type**: `string` **Default**: `redpanda-connect` ### [](#default_ttl)`default_ttl` An optional default TTL to set for items, calculated from the moment the item is cached. **Type**: `string` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#prefix)`prefix` An optional string to prefix item keys with in order to prevent collisions with similar services. **Type**: `string` ### [](#retries)`retries` Determine time intervals and cut offs for retry attempts. **Type**: `object` ### [](#retries-initial_interval)`retries.initial_interval` The initial period to wait between retry attempts. **Type**: `string` **Default**: `500ms` ```yaml # Examples: initial_interval: 50ms # --- initial_interval: 1s ``` ### [](#retries-max_elapsed_time)`retries.max_elapsed_time` The maximum overall period of time to spend on retry attempts before the request is aborted. **Type**: `string` **Default**: `5s` ```yaml # Examples: max_elapsed_time: 1m # --- max_elapsed_time: 1h ``` ### [](#retries-max_interval)`retries.max_interval` The maximum period to wait between retry attempts **Type**: `string` **Default**: `1s` ```yaml # Examples: max_interval: 5s # --- max_interval: 1m ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 29: redpanda **URL**: https://docs.redpanda.com/redpanda-connect/components/caches/redpanda.md --- # redpanda --- title: redpanda latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: caches/redpanda page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/redpanda.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/redpanda.adoc description: A Kafka cache using the https://github.com/twmb/franz-go[Franz Kafka client library^]. page-git-created-date: "2025-06-25" page-git-modified-date: "2025-10-06" --- **Type:** Cache ▼ [Cache](/redpanda-connect/components/caches/redpanda/)[Input](/redpanda-connect/components/inputs/redpanda/)[Output](/redpanda-connect/components/outputs/redpanda/)[Tracer](/redpanda-connect/components/tracers/redpanda/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/caches/redpanda/ "View the Cloud version of this component"), Self-Managed A Kafka cache implemented using the [Franz Kafka client library](https://github.com/twmb/franz-go). #### Common ```yaml caches: redpanda: seed_brokers: [] # No default (required) topic: "" # No default (required) ``` #### Advanced ```yaml caches: redpanda: seed_brokers: [] # No default (required) client_id: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) metadata_max_age: 1m request_timeout_overhead: 10s conn_idle_timeout: 20s tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s topic: "" # No default (required) allow_auto_topic_creation: true ``` A cache that stores data in a Kafka topic. This cache is useful for data that is written frequently and queried infrequently. Reads from the cache require scanning the entire topic partition. If you expect frequent access, consider placing an in-memory caching layer in front of this one. Because only the latest values are needed, configure compaction for topics used as caches so that reads are less expensive when topics are rescanned. See [Compaction Settings](../../../../current/manage/cluster-maintenance/compaction-settings/). The cache does not have any TTL mechanisms. Use the Kafka topic retention policies to manage TTL. ## [](#fields)Fields ### [](#allow_auto_topic_creation)`allow_auto_topic_creation` Enables topics to be auto created if they do not exist when fetching their metadata. **Type**: `bool` **Default**: `true` ### [](#client_id)`client_id` An identifier for the client connection. **Type**: `string` **Default**: `redpanda-connect` ### [](#conn_idle_timeout)`conn_idle_timeout` The amount of time that connections can remain idle before they are closed. **Type**: `string` **Default**: `20s` ### [](#metadata_max_age)`metadata_max_age` The maximum age of metadata before it is refreshed. This interval also controls how frequently regex topic patterns are re-evaluated to discover new matching topics. **Type**: `string` **Default**: `1m` ### [](#request_timeout_overhead)`request_timeout_overhead` Additional time to apply as overhead when calculating request deadlines. This buffer helps prevent premature timeouts, especially for requests that already define their own timeout values. **Type**: `string` **Default**: `10s` ### [](#sasl)`sasl[]` Specify one or more SASL authentication methods. Each method is tried in the order specified. If the broker supports the first mechanism, outgoing client connections use that mechanism. If the first mechanism fails, the client will use the first supported mechanism. If the broker does not support any client mechanisms, connections will fail. **Type**: `object` ```yaml # Examples: sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ### [](#sasl-aws)`sasl[].aws` Contains AWS-specific fields for when [`sasl.mechanism`](#sasl-mechanism) is set to `AWS_MSK_IAM`. **Type**: `object` ### [](#sasl-aws-credentials)`sasl[].aws.credentials` Optional manual configuration of AWS credentials to use. For more information, see the [credentials for AWS](../../../guides/cloud/aws/) guide. **Type**: `object` ### [](#sasl-aws-credentials-from_ec2_role)`sasl[].aws.credentials.from_ec2_role` The credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#sasl-aws-credentials-id)`sasl[].aws.credentials.id` The ID of credentials to use. **Type**: `string` ### [](#sasl-aws-credentials-profile)`sasl[].aws.credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#sasl-aws-credentials-role)`sasl[].aws.credentials.role` The ARN of the role to assume. **Type**: `string` ### [](#sasl-aws-credentials-role_external_id)`sasl[].aws.credentials.role_external_id` An external ID to provide when assuming the specified role. **Type**: `string` ### [](#sasl-aws-credentials-secret)`sasl[].aws.credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#sasl-aws-credentials-token)`sasl[].aws.credentials.token` The token for the credentials being used. Required only when using short-term credentials. **Type**: `string` ### [](#sasl-aws-endpoint)`sasl[].aws.endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#sasl-aws-region)`sasl[].aws.region` The AWS region to target. **Type**: `string` ### [](#sasl-aws-tcp)`sasl[].aws.tcp` TCP socket configuration. **Type**: `object` ### [](#sasl-aws-tcp-connect_timeout)`sasl[].aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-aws-tcp-keep_alive)`sasl[].aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#sasl-aws-tcp-keep_alive-count)`sasl[].aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#sasl-aws-tcp-keep_alive-idle)`sasl[].aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-keep_alive-interval)`sasl[].aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-tcp_user_timeout)`sasl[].aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-extensions)`sasl[].extensions` Key/value pairs to add to OAUTHBEARER authentication requests. **Type**: `string` ### [](#sasl-mechanism)`sasl[].mechanism` The SASL mechanism to use for authentication. **Type**: `string` | Option | Summary | | --- | --- | | AWS_MSK_IAM | AWS IAM-based authentication as specified by the aws-msk-iam-auth Java library. | | OAUTHBEARER | OAuth Bearer authentication. | | PLAIN | PLAIN mechanism for plaintext password authentication. | | REDPANDA_CLOUD_SERVICE_ACCOUNT | Redpanda Cloud Service Account authentication when running in Redpanda Cloud. | | SCRAM-SHA-256 | SCRAM authentication as specified in RFC5802. | | SCRAM-SHA-512 | SCRAM authentication as specified in RFC5802. | | none | Disable SASL authentication. | ### [](#sasl-password)`sasl[].password` The password to use for PLAIN or SCRAM-\* authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#sasl-token)`sasl[].token` The token to use for a single session’s OAUTHBEARER authentication. **Type**: `string` **Default**: `""` ### [](#sasl-username)`sasl[].username` The username to use for PLAIN or SCRAM-\* authentication. **Type**: `string` **Default**: `""` ### [](#seed_brokers)`seed_brokers[]` A list of broker addresses to connect to. Items containing commas are expanded into multiple addresses. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" # --- seed_brokers: - "foo:9092" - "bar:9092" # --- seed_brokers: - "foo:9092,bar:9092" ``` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic)`topic` The topic to store data in. **Type**: `string` --- # Page 30: ristretto **URL**: https://docs.redpanda.com/redpanda-connect/components/caches/ristretto.md --- # ristretto --- title: ristretto latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: caches/ristretto page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/ristretto.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/ristretto.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/caches/ristretto/ "View the Cloud version of this component"), Self-Managed Stores key/value pairs in a map held in the memory-bound [Ristretto cache](https://github.com/dgraph-io/ristretto). #### Common ```yml caches: ristretto: default_ttl: "" ``` #### Advanced ```yml caches: ristretto: default_ttl: "" get_retries: enabled: false initial_interval: 1s max_interval: 5s max_elapsed_time: 30s ``` This cache is more efficient and appropriate for high-volume use cases than the standard memory cache. However, the add command is non-atomic, and therefore this cache is not suitable for deduplication. ## [](#fields)Fields ### [](#default_ttl)`default_ttl` A default TTL to set for items, calculated from the moment the item is cached. Set to an empty string or zero duration to disable TTLs. **Type**: `string` **Default**: `""` ```yaml # Examples: default_ttl: 5m # --- default_ttl: 60s ``` ### [](#get_retries)`get_retries` Determines how and whether get attempts should be retried if the key is not found. Ristretto is a concurrent cache that does not immediately reflect writes, and so it can sometimes be useful to enable retries at the cost of speed in cases where the key is expected to exist. **Type**: `object` ### [](#get_retries-enabled)`get_retries.enabled` Whether retries should be enabled. **Type**: `bool` **Default**: `false` ### [](#get_retries-initial_interval)`get_retries.initial_interval` The initial period to wait between retry attempts. **Type**: `string` **Default**: `1s` ```yaml # Examples: initial_interval: 50ms # --- initial_interval: 1s ``` ### [](#get_retries-max_elapsed_time)`get_retries.max_elapsed_time` The maximum overall period of time to spend on retry attempts before the request is aborted. **Type**: `string` **Default**: `30s` ```yaml # Examples: max_elapsed_time: 1m # --- max_elapsed_time: 1h ``` ### [](#get_retries-max_interval)`get_retries.max_interval` The maximum period to wait between retry attempts **Type**: `string` **Default**: `5s` ```yaml # Examples: max_interval: 5s # --- max_interval: 1m ``` --- # Page 31: sql **URL**: https://docs.redpanda.com/redpanda-connect/components/caches/sql.md --- # sql --- title: sql latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: caches/sql page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/sql.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/sql.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Cache ▼ [Cache](/redpanda-connect/components/caches/sql/)[Output](/redpanda-connect/components/outputs/sql/)[Processor](/redpanda-connect/components/processors/sql/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/caches/sql/ "View the Cloud version of this component"), Self-Managed Uses an SQL database table as a destination for storing cache key/value items. Introduced in version 4.26.0. #### Common ```yml caches: sql: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) key_column: "" # No default (required) value_column: "" # No default (required) set_suffix: "" # No default (optional) ``` #### Advanced ```yml caches: sql: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) key_column: "" # No default (required) value_column: "" # No default (required) set_suffix: "" # No default (optional) init_files: [] # No default (optional) init_statement: "" # No default (optional) conn_max_idle_time: "" # No default (optional) conn_max_life_time: "" # No default (optional) conn_max_idle: 2 conn_max_open: "" # No default (optional) ``` Each cache key/value pair will exist as a row within the specified table. Currently only the key and value columns are set, and therefore any other columns present within the target table must allow NULL values if this cache is going to be used for set and add operations. Cache operations are translated into SQL statements as follows: ## [](#get)Get All `get` operations are performed with a traditional `select` statement. ## [](#delete)Delete All `delete` operations are performed with a traditional `delete` statement. ## [](#set)Set The `set` operation is performed with a traditional `insert` statement. This will behave as an `add` operation by default, and so ideally needs to be adapted in order to provide updates instead of failing on collision s. Since different SQL engines implement upserts differently it is necessary to specify a `set_suffix` that modifies an `insert` statement in order to perform updates on conflict. ## [](#add)Add The `add` operation is performed with a traditional `insert` statement. ## [](#fields)Fields ### [](#conn_max_idle)`conn_max_idle` An optional maximum number of connections in the idle connection pool. If conn\_max\_open is greater than 0 but less than the new conn\_max\_idle, then the new conn\_max\_idle will be reduced to match the conn\_max\_open limit. If `value ⇐ 0`, no idle connections are retained. The default max idle connections is currently 2. This may change in a future release. **Type**: `int` **Default**: `2` ### [](#conn_max_idle_time)`conn_max_idle_time` An optional maximum amount of time a connection may be idle. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections idle time. **Type**: `string` ### [](#conn_max_life_time)`conn_max_life_time` An optional maximum amount of time a connection may be reused. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections age. **Type**: `string` ### [](#conn_max_open)`conn_max_open` An optional maximum number of open connections to the database. If conn\_max\_idle is greater than 0 and the new conn\_max\_open is less than conn\_max\_idle, then conn\_max\_idle will be reduced to match the new conn\_max\_open limit. If `value ⇐ 0`, then there is no limit on the number of open connections. The default is 0 (unlimited). **Type**: `int` ### [](#driver)`driver` A database [driver](#drivers) to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks` ### [](#dsn)`dsn` A Data Source Name to identify the target database. #### [](#drivers)Drivers The following is a list of supported drivers, their placeholder style, and their respective DSN formats: | Driver | Data Source Name Format | | --- | --- | | clickhouse | clickhouse://[username[:password]@][netloc][:port]/dbname[?param1=value1&…​¶mN=valueN] | | mysql | [username[:password]@][protocol[(address)]]/dbname[?param1=value1&…​¶mN=valueN] | | postgres and pgx | postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&…​] | | mssql | sqlserver://[user[:password]@][netloc][:port][?database=dbname¶m1=value1&…​] | | sqlite | file:/path/to/filename.db[?param&=value1&…​] | | oracle | oracle://[username[:password]@][netloc][:port]/service_name?server=server2&server=server3 | | snowflake | username[:password]@account_identifier/dbname/schemaname[?param1=value&…​¶mN=valueN] | | trino | http[s]://user[:pass]@host[:port][?parameters] | | gocosmos | AccountEndpoint=;AccountKey=[;TimeoutMs=][;Version=][;DefaultDb/Db=][;AutoId=][;InsecureSkipVerify=] | | spanner | projects/[PROJECT]/instances/[INSTANCE]/databases/[DATABASE] | | databricks | token:@:/ | Please note that the `postgres` and `pgx` drivers enforce SSL by default, you can override this with the parameter `sslmode=disable` if required. The `pgx` driver is an alternative to the standard `postgres` (pq) driver and comes with extra functionality such as support for array insertion. The `snowflake` driver supports multiple DSN formats. Please consult [the docs](https://pkg.go.dev/github.com/snowflakedb/gosnowflake#hdr-Connection_String) for more details. For [key pair authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication), the DSN has the following format: `@//?warehouse=&role=&authenticator=snowflake_jwt&privateKey=`, where the value for the `privateKey` parameter can be constructed from an unencrypted RSA private key file `rsa_key.p8` using `openssl enc -d -base64 -in rsa_key.p8 | basenc --base64url -w0` (you can use `gbasenc` instead of `basenc` on OSX if you install `coreutils` via Homebrew). If you have a password-encrypted private key, you can decrypt it using `openssl pkcs8 -in rsa_key_encrypted.p8 -out rsa_key.p8`. Also, make sure fields such as the username are URL-encoded. The [`gocosmos`](https://pkg.go.dev/github.com/microsoft/gocosmos) driver is still experimental, but it has support for [hierarchical partition keys](https://learn.microsoft.com/en-us/azure/cosmos-db/hierarchical-partition-keys) as well as [cross-partition queries](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-query-container#cross-partition-query). Please refer to the [SQL notes](https://github.com/microsoft/gocosmos/blob/main/SQL.md) for details. **Type**: `string` ```yaml # Examples: dsn: clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60 # --- dsn: foouser:foopassword@tcp(localhost:3306)/foodb # --- dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable # --- dsn: oracle://foouser:foopass@localhost:1521/service_name # --- dsn: token:dapi1234567890ab@dbc-a1b2345c-d6e7.cloud.databricks.com:443/sql/1.0/warehouses/abc123def456 ``` ### [](#init_files)`init_files[]` An optional list of file paths containing SQL statements to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Glob patterns are supported, including super globs (double star). Care should be taken to ensure that the statements are idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If a statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `array` ```yaml # Examples: init_files: - ./init/*.sql # --- init_files: - ./foo.sql - ./bar.sql ``` ### [](#init_statement)`init_statement` An optional SQL statement to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Care should be taken to ensure that the statement is idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If the statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `string` ```yaml # Examples: init_statement: |- CREATE TABLE IF NOT EXISTS some_table ( foo varchar(50) not null, bar integer, baz varchar(50), primary key (foo) ) WITHOUT ROWID; ``` ### [](#key_column)`key_column` The name of a column to be used for storing cache item keys. This column should support strings of arbitrary size. **Type**: `string` ```yaml # Examples: key_column: foo ``` ### [](#set_suffix)`set_suffix` An optional suffix to append to each insert query for a cache `set` operation. This should modify an insert statement into an upsert appropriate for the given SQL engine. **Type**: `string` ```yaml # Examples: set_suffix: ON DUPLICATE KEY UPDATE bar=VALUES(bar) # --- set_suffix: ON CONFLICT (foo) DO UPDATE SET bar=excluded.bar # --- set_suffix: ON CONFLICT (foo) DO NOTHING ``` ### [](#table)`table` The table to insert/read/delete cache items. **Type**: `string` ```yaml # Examples: table: foo ``` ### [](#value_column)`value_column` The name of a column to be used for storing cache item values. This column should support strings of arbitrary size. **Type**: `string` ```yaml # Examples: value_column: bar ``` --- # Page 32: ttlru **URL**: https://docs.redpanda.com/redpanda-connect/components/caches/ttlru.md --- # ttlru --- title: ttlru latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: caches/ttlru page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: caches/ttlru.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/caches/ttlru.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-03-13" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/caches/ttlru/ "View the Cloud version of this component"), Self-Managed Stores key/value pairs in a ttlru in-memory cache. This cache is therefore reset every time the service restarts. #### Common ```yml caches: ttlru: cap: 1024 default_ttl: 5m0s init_values: {} ``` #### Advanced ```yml caches: ttlru: cap: 1024 default_ttl: 5m0s ttl: "" # No default (optional) init_values: {} optimistic: false ``` The cache ttlru provides a simple, goroutine safe, cache with a fixed number of entries. Each entry has a per-cache defined TTL. This TTL is reset on both modification and access of the value. As a result, if the cache is full, and no items have expired, when adding a new item, the item with the soonest expiration will be evicted. It uses the package [`expirable`](https://github.com/hashicorp/golang-lru/tree/main/expirable) The field init\_values can be used to pre-populate the memory cache with any number of key/value pairs: ```yaml cache_resources: - label: foocache ttlru: default_ttl: '5m' cap: 1024 init_values: foo: bar ``` These values can be overridden during execution. ## [](#fields)Fields ### [](#cap)`cap` The cache maximum capacity (number of entries) **Type**: `int` **Default**: `1024` ### [](#default_ttl)`default_ttl` The cache ttl of each element Requires version 4.21.0 or later. **Type**: `string` **Default**: `5m0s` ### [](#init_values)`init_values` A table of key/value pairs that should be present in the cache on initialization. This can be used to create static lookup tables. **Type**: `string` **Default**: `{}` ```yaml # Examples: init_values: Nickelback: "1995" Spice Girls: "1994" The Human League: "1977" ``` ### [](#optimistic)`optimistic` If true, we do not lock on read/write events. The ttlru package is thread-safe, however the ADD operation is not atomic. **Type**: `bool` **Default**: `false` ### [](#ttl)`ttl` Deprecated. Please use `default_ttl` field **Type**: `string` --- # Page 33: HTTP **URL**: https://docs.redpanda.com/redpanda-connect/components/http/about.md --- # HTTP --- title: HTTP latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: http/about page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: http/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/http/about.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- When Redpanda Connect runs it kicks off an HTTP server that provides a few generally useful endpoints and is also where configured components such as the [`http_server` input](../../inputs/http_server/) [and output](../../outputs/http_server/) can register their own endpoints if they don’t require their own host/port. The configuration for this server lives under the `http` namespace, with the following default values: #### Common ```yaml # Common config fields, showing default values http: enabled: true address: 0.0.0.0:4195 root_path: /benthos debug_endpoints: false ``` #### Advanced ```yaml # All config fields, showing default values http: enabled: true address: 0.0.0.0:4195 root_path: /benthos debug_endpoints: false cert_file: "" key_file: "" cors: enabled: false allowed_origins: [] basic_auth: enabled: false realm: restricted username: "" password_hash: "" algorithm: sha256 salt: "" ``` The field `enabled` can be set to `false` in order to disable the server. The field `root_path` specifies a general prefix for all endpoints, this can help isolate the service endpoints when using a reverse proxy with other shared services. All endpoints will still be registered at the root as well as behind the prefix, e.g. with a `root_path` set to `/foo` the endpoint `/version` will be accessible from both `/version` and `/foo/version`. ## [](#enabling-https)Enabling HTTPS By default Redpanda Connect will serve traffic over HTTP. In order to enforce TLS and serve traffic exclusively over HTTPS you must provide a `cert_file` and `key_file` path in your config, which point to a file containing a certificate and a matching private key for the server respectively. If the certificate is signed by a certificate authority, the `cert_file` should be the concatenation of the server’s certificate, any intermediates, and the CA’s certificate. ## [](#enabling-basic-authentication)Enabling basic authentication By default Redpanda Connect does not do any sort of authentication for the service-wide HTTP server. However, it’s possible to configure basic authentication with the [`basic_auth`](#basic-auth) field. Passwords configured must be hashed according to the specified algorithm and base64 encoded, for some hashing algorithms you can do this using Redpanda Connect itself: ```sh echo mynewpassword | rpk connect blobl 'root = content().hash("sha256").encode("base64")' ``` ## [](#endpoints)Endpoints The following endpoints will be generally available when the HTTP server is enabled: - `/version` provides version info. - `/ping` can be used as a liveness probe as it always returns a 200. - `/ready` can be used as a readiness probe as it serves a 200 only when both the input and output are connected, otherwise a 503 is returned. - `/metrics`, `/stats` both provide metrics when the metrics type is either [`json_api`](../../metrics/json_api/) or [`prometheus`](../../metrics/prometheus/). - `/endpoints` provides a JSON object containing a list of available endpoints, including those registered by configured components. ## [](#cors)CORS In order to serve Cross-Origin Resource Sharing headers, which instruct browsers to allow CORS requests, set the subfield `cors.enabled` to `true`. ### [](#allowed_origins)allowed\_origins A list of allowed origins to connect from. The literal value `*` can be specified as a wildcard. Note `cors.enabled` must be set to `true` for this list to take effect. ## [](#debug-endpoints)Debug endpoints The field `debug_endpoints` when set to `true` prompts Redpanda Connect to register a few extra endpoints that can be useful for debugging performance or behavioral problems: - `/debug/config/json` returns the loaded config as JSON. - `/debug/config/yaml` returns the loaded config as YAML. - `/debug/pprof/block` responds with a pprof-formatted block profile. - `/debug/pprof/heap` responds with a pprof-formatted heap profile. - `/debug/pprof/mutex` responds with a pprof-formatted mutex profile. - `/debug/pprof/profile` responds with a pprof-formatted cpu profile. - `/debug/pprof/goroutine` responds with a pprof-formatted goroutine profile. - `/debug/pprof/symbol` looks up the program counters listed in the request, responding with a table mapping program counters to function names. - `/debug/pprof/trace` responds with the execution trace in binary form. Tracing lasts for duration specified in seconds GET parameter, or for 1 second if not specified. - `/debug/stack` returns a snapshot of the current service stack trace. ## [](#fields)Fields The schema of the `http` section is as follows: ### [](#enabled)`enabled` Whether to enable to HTTP server. **Type**: `bool` **Default**: `true` ### [](#address)`address` The address to bind to. **Type**: `string` **Default**: `"0.0.0.0:4195"` ### [](#root_path)`root_path` Specifies a general prefix for all endpoints, this can help isolate the service endpoints when using a reverse proxy with other shared services. All endpoints will still be registered at the root as well as behind the prefix, e.g. with a root\_path set to `/foo` the endpoint `/version` will be accessible from both `/version` and `/foo/version`. **Type**: `string` **Default**: `"/benthos"` ### [](#debug_endpoints)`debug_endpoints` Whether to register a few extra endpoints that can be useful for debugging performance or behavioral problems. **Type**: `bool` **Default**: `false` ### [](#cert_file)`cert_file` An optional certificate file for enabling TLS. **Type**: `string` **Default**: `""` ### [](#key_file)`key_file` An optional key file for enabling TLS. **Type**: `string` **Default**: `""` ### [](#cors-2)`cors` Adds Cross-Origin Resource Sharing headers. **Type**: `object` Requires version 3.63.0 or newer ### [](#cors-enabled)`cors.enabled` Whether to allow CORS requests. **Type**: `bool` **Default**: `false` ### [](#cors-allowed_origins)`cors.allowed_origins` An explicit list of origins that are allowed for CORS requests. **Type**: `array` **Default**: `[]` ### [](#basic_auth)`basic_auth` Allows you to enforce and customise basic authentication for requests to the HTTP server. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Enable basic authentication **Type**: `bool` **Default**: `false` ### [](#basic_auth-realm)`basic_auth.realm` Custom realm name **Type**: `string` **Default**: `"restricted"` ### [](#basic_auth-username)`basic_auth.username` Username required to authenticate. **Type**: `string` **Default**: `""` ### [](#basic_auth-password_hash)`basic_auth.password_hash` Hashed password required to authenticate. (base64 encoded) **Type**: `string` **Default**: `""` ### [](#basic_auth-algorithm)`basic_auth.algorithm` Encryption algorithm used to generate `password_hash`. **Type**: `string` **Default**: `"sha256"` ```yml # Examples algorithm: md5 algorithm: sha256 algorithm: bcrypt algorithm: scrypt ``` ### [](#basic_auth-salt)`basic_auth.salt` Salt for scrypt algorithm. (base64 encoded) **Type**: `string` **Default**: `""` --- # Page 34: Inputs **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/about.md --- # Inputs --- title: Inputs latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/about page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/about.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-09" --- An input is a source of data piped through an array of optional [processors](../../processors/about/): ```yaml input: label: my_redis_input redis_streams: url: tcp://localhost:6379 streams: - benthos_stream body_key: body consumer_group: benthos_group # Optional list of processing steps processors: - mapping: | root.document = this.without("links") root.link_count = this.links.length() ``` Some inputs have a logical end, for example a [`csv` input](../csv/) ends once the last row is consumed, when this happens the input gracefully terminates and Redpanda Connect will shut itself down once all messages have been processed fully. It’s also possible to specify a logical end for an input that otherwise doesn’t have one with the [`read_until` input](../read_until/), which checks a condition against each consumed message in order to determine whether it should be the last. ## [](#brokering)Brokering Only one input is configured at the root of a Redpanda Connect config. However, the root input can be a [broker](../broker/) which combines multiple inputs and merges the streams: ```yaml input: broker: inputs: - kafka: addresses: [ TODO ] topics: [ foo, bar ] consumer_group: foogroup - redis_streams: url: tcp://localhost:6379 streams: - benthos_stream body_key: body consumer_group: benthos_group ``` ## [](#labels)Labels Inputs have an optional field `label` that can uniquely identify them in observability data such as metrics and logs. This can be useful when running configs with multiple inputs, otherwise their metrics labels will be generated based on their composition. For more information check out the [metrics documentation](../../metrics/about/). ### [](#sequential-reads)Sequential reads Sometimes it’s useful to consume a sequence of inputs, where an input is only consumed once its predecessor is drained fully, you can achieve this with the [`sequence` input](../sequence/). ## [](#generating-messages)Generating messages It’s possible to generate data with Redpanda Connect using the [`generate` input](../generate/), which is also a convenient way to trigger scheduled pipelines. ## [](#categories)Categories ### Services Inputs that consume from storage or message streaming services. [**RabbitMQ**](/redpanda-connect/components/inputs/amqp_0_9/) [**RabbitMQ**](/redpanda-connect/components/inputs/amqp_1/) [**Amazon DynamoDB CDC**](/redpanda-connect/components/inputs/aws_dynamodb_cdc/) [**AWS CloudWatch Logs**](/redpanda-connect/components/inputs/aws_cloudwatch_logs/) [**AWS Kinesis**](/redpanda-connect/components/inputs/aws_kinesis/) [**AWS S3**](/redpanda-connect/components/inputs/aws_s3/) [**AWS SQS**](/redpanda-connect/components/inputs/aws_sqs/) [**Azure Blob Storage**](/redpanda-connect/components/inputs/azure_blob_storage/) [**Azure Queue Storage**](/redpanda-connect/components/inputs/azure_queue_storage/) [**Azure Table Storage**](/redpanda-connect/components/inputs/azure_table_storage/) [**beanstalkd**](/redpanda-connect/components/inputs/beanstalkd/) [**Apache Cassandra**](/redpanda-connect/components/inputs/cassandra/) [**discord**](/redpanda-connect/components/inputs/discord/) [**GCP BigQuery**](/redpanda-connect/components/inputs/gcp_bigquery_select/) [**GCP Cloud Storage**](/redpanda-connect/components/inputs/gcp_cloud_storage/) [**GCP PubSub**](/redpanda-connect/components/inputs/gcp_pubsub/) [**Google Cloud**](/redpanda-connect/components/inputs/gcp_spanner_cdc/) [**git**](/redpanda-connect/components/inputs/git/) [**hdfs**](/redpanda-connect/components/inputs/hdfs/) [**microsoft\_sql\_server\_cdc**](/redpanda-connect/components/inputs/microsoft_sql_server_cdc/) [**MongoDB**](/redpanda-connect/components/inputs/mongodb/) [**MongoDB CDC**](/redpanda-connect/components/inputs/mongodb_cdc/) [**mqtt**](/redpanda-connect/components/inputs/mqtt/) [**mysql\_cdc**](/redpanda-connect/components/inputs/mysql_cdc/) [**NATS**](/redpanda-connect/components/inputs/nats/) [**NATS KV**](/redpanda-connect/components/inputs/nats_kv/) [**NATS JetStream**](/redpanda-connect/components/inputs/nats_jetstream/) [**NATS Stream**](/redpanda-connect/components/inputs/nats_stream/) [**nsq**](/redpanda-connect/components/inputs/nsq/) [**ockam\_kafka**](/redpanda-connect/components/inputs/ockam_kafka/) [**Oracle CDC**](/redpanda-connect/components/inputs/oracledb_cdc/) [**OpenTelemetry**](/redpanda-connect/components/inputs/otlp_grpc/) [**OpenTelemetry**](/redpanda-connect/components/inputs/otlp_http/) [**postgres\_cdc**](/redpanda-connect/components/inputs/postgres_cdc/) [**Apache Pulsar**](/redpanda-connect/components/inputs/pulsar/) [**Redis List**](/redpanda-connect/components/inputs/redis_list/) [**Redis PubSub**](/redpanda-connect/components/inputs/redis_pubsub/) [**Redis**](/redpanda-connect/components/inputs/redis_scan/) [**Redis Streams**](/redpanda-connect/components/inputs/redis_streams/) [**redpanda**](/redpanda-connect/components/inputs/redpanda/) [**redpanda\_common**](/redpanda-connect/components/inputs/redpanda_common/) [**redpanda\_migrator**](/redpanda-connect/components/inputs/redpanda_migrator/) [**spicedb\_watch**](/redpanda-connect/components/inputs/spicedb_watch/) [**splunk**](/redpanda-connect/components/inputs/splunk/) [**SQL**](/redpanda-connect/components/inputs/sql_raw/) [**SQL**](/redpanda-connect/components/inputs/sql_select/) [**tigerbeetle\_cdc**](/redpanda-connect/components/inputs/tigerbeetle_cdc/) [**timeplus**](/redpanda-connect/components/inputs/timeplus/) [**twitter\_search**](/redpanda-connect/components/inputs/twitter_search/) ### AWS Inputs that consume from Amazon Web Services products. [**AWS CloudWatch Logs**](/redpanda-connect/components/inputs/aws_cloudwatch_logs/) [**AWS Kinesis**](/redpanda-connect/components/inputs/aws_kinesis/) [**AWS S3**](/redpanda-connect/components/inputs/aws_s3/) [**AWS SQS**](/redpanda-connect/components/inputs/aws_sqs/) ### Azure Inputs that consume from Microsoft Azure services. [**Azure Blob Storage**](/redpanda-connect/components/inputs/azure_blob_storage/) [**Microsoft Azure**](/redpanda-connect/components/inputs/azure_cosmosdb/) [**Azure Queue Storage**](/redpanda-connect/components/inputs/azure_queue_storage/) [**Azure Table Storage**](/redpanda-connect/components/inputs/azure_table_storage/) ### Utility Inputs that provide utility by generating data or combining/wrapping other inputs. [**batched**](/redpanda-connect/components/inputs/batched/) [**broker**](/redpanda-connect/components/inputs/broker/) [**dynamic**](/redpanda-connect/components/inputs/dynamic/) [**generate**](/redpanda-connect/components/inputs/generate/) [**inproc**](/redpanda-connect/components/inputs/inproc/) [**read\_until**](/redpanda-connect/components/inputs/read_until/) [**resource**](/redpanda-connect/components/inputs/resource/) [**sequence**](/redpanda-connect/components/inputs/sequence/) [**subprocess**](/redpanda-connect/components/inputs/subprocess/) ### Integration [**cockroachdb\_changefeed**](/redpanda-connect/components/inputs/cockroachdb_changefeed/) [**schema\_registry**](/redpanda-connect/components/inputs/schema_registry/) ### Local Inputs that consume from the local machine/filesystem. [**CSV**](/redpanda-connect/components/inputs/csv/) [**file**](/redpanda-connect/components/inputs/file/) [**parquet**](/redpanda-connect/components/inputs/parquet/) [**stdin**](/redpanda-connect/components/inputs/stdin/) ### Social Inputs that consume from social applications and services. [**discord**](/redpanda-connect/components/inputs/discord/) [**twitter\_search**](/redpanda-connect/components/inputs/twitter_search/) ### GCP Inputs that consume from Google Cloud Platform services. [**GCP BigQuery**](/redpanda-connect/components/inputs/gcp_bigquery_select/) [**GCP Cloud Storage**](/redpanda-connect/components/inputs/gcp_cloud_storage/) [**GCP PubSub**](/redpanda-connect/components/inputs/gcp_pubsub/) [**Google Cloud**](/redpanda-connect/components/inputs/gcp_spanner_cdc/) ### Network Inputs that consume directly from low level network protocols. [**HTTP**](/redpanda-connect/components/inputs/http_client/) [**HTTP**](/redpanda-connect/components/inputs/http_server/) [**nanomsg**](/redpanda-connect/components/inputs/nanomsg/) [**OpenTelemetry**](/redpanda-connect/components/inputs/otlp_grpc/) [**OpenTelemetry**](/redpanda-connect/components/inputs/otlp_http/) [**sftp**](/redpanda-connect/components/inputs/sftp/) [**socket**](/redpanda-connect/components/inputs/socket/) [**socket\_server**](/redpanda-connect/components/inputs/socket_server/) [**websocket**](/redpanda-connect/components/inputs/websocket/) [**zmq4**](/redpanda-connect/components/inputs/zmq4/) --- # Page 35: amqp_0_9 **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/amqp_0_9.md --- # amqp\_0\_9 --- title: amqp_0_9 latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/amqp_0_9 page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/amqp_0_9.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/amqp_0_9.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/amqp_0_9/)[Output](/redpanda-connect/components/outputs/amqp_0_9/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/amqp_0_9/ "View the Cloud version of this component"), Self-Managed Connects to an AMQP (0.91) queue. AMQP is a messaging protocol used by various message brokers, including RabbitMQ. #### Common ```yml inputs: label: "" amqp_0_9: urls: [] # No default (required) queue: "" # No default (required) consumer_tag: "" prefetch_count: 10 ``` #### Advanced ```yml inputs: label: "" amqp_0_9: urls: [] # No default (required) queue: "" # No default (required) queue_declare: enabled: false durable: true auto_delete: false arguments: "" # No default (optional) bindings_declare: [] # No default (optional) consumer_tag: "" auto_ack: false nack_reject_patterns: [] prefetch_count: 10 prefetch_size: 0 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] ``` TLS is automatically enabled when connecting to an `amqps` URL. However, you can customize [TLS settings](#tls) if required. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `amqp_content_type` - `amqp_content_encoding` - `amqp_delivery_mode` - `amqp_priority` - `amqp_correlation_id` - `amqp_reply_to` - `amqp_expiration` - `amqp_message_id` - `amqp_timestamp` - `amqp_type` - `amqp_user_id` - `amqp_app_id` - `amqp_consumer_tag` - `amqp_delivery_tag` - `amqp_redelivered` - `amqp_exchange` - `amqp_routing_key` - All existing message headers, including nested headers prefixed with the key of their respective parent. You can access these metadata fields using [function interpolations](../../../configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#auto_ack)`auto_ack` Set to `true` to automatically acknowledge messages as soon as they are consumed rather than waiting for acknowledgments from downstream. This can improve throughput and prevent the pipeline from becoming blocked, but delivery guarantees are lost. **Type**: `bool` **Default**: `false` ### [](#bindings_declare)`bindings_declare[]` Passively declares the bindings of the target queue to make sure they exist and are configured correctly. If the bindings exist, then the passive declaration verifies that fields specified in this object match them. **Type**: `object` ```yaml # Examples: bindings_declare: - exchange: foo key: bar ``` ### [](#bindings_declare-exchange)`bindings_declare[].exchange` The exchange of the declared binding. **Type**: `string` **Default**: `""` ### [](#bindings_declare-key)`bindings_declare[].key` The key of the declared binding. **Type**: `string` **Default**: `""` ### [](#consumer_tag)`consumer_tag` A consumer tag to uniquely identify the consumer. **Type**: `string` **Default**: `""` ### [](#nack_reject_patterns)`nack_reject_patterns[]` A list of regular expression patterns to match against errors in messages that Redpanda Connect fails to deliver. When a message has an error that matches a pattern, it is dropped or delivered to a dead-letter queue (if a queue has been configured). By default, failed messages are negatively acknowledged (nacked) and requeued. Requires version 3.64.0 or later. **Type**: `array` **Default**: `[]` ```yaml # Examples: nack_reject_patterns: - "^reject me please:.+$" ``` ### [](#prefetch_count)`prefetch_count` The maximum number of pending messages at a given time. **Type**: `int` **Default**: `10` ### [](#prefetch_size)`prefetch_size` The maximum size of pending messages (in bytes) at a given time. **Type**: `int` **Default**: `0` ### [](#queue)`queue` An AMQP queue to consume from. **Type**: `string` ### [](#queue_declare)`queue_declare` Passively declares the [target queue](#queue) to make sure a queue with the specified name exists and is configured correctly. If the queue exists, then the passive declaration verifies that fields specified in this object match the its properties. **Type**: `object` ### [](#queue_declare-arguments)`queue_declare.arguments` Arguments for server-specific implementations of the queue (optional). You can use arguments to configure additional parameters for queue types that require them. For more information about available arguments, see the [RabbitMQ Client Library](https://github.com/rabbitmq/amqp091-go/blob/b3d409fe92c34bea04d8123a136384c85e8dc431/types.go#L282-L362). | Argument | Description | Accepted values | | --- | --- | --- | | x-queue-type | Declares the type of queue. | Options: classic (default), quorum, stream, drop-head, reject-publish, and reject-publish-dlx. | | x-max-length | The maximum number of messages in the queue. | A non-negative integer. | | x-max-length-bytes | The maximum size of messages (in bytes) in the queue. | A non-negative integer. | | x-overflow | Sets the queue’s overflow behavior. | Options: drop-head (default), reject-publish, reject-publish-dlx. | | x-message-ttl | The duration (in milliseconds) that messages remain in the queue before they expire and are discarded. | A string that represents the number of milliseconds. For example, 60000 retains messages for one minute. | | x-expires | The duration after which the queue automatically expires. | A positive integer. | | x-max-age | The duration (in configurable units) that streamed messages are retained on disk before they are discarded. | Options: Y, M, D, h, m, s. For example, 7D retains messages for a week. | | x-stream-max-segment-size-bytes | The maximum size (in bytes) of the segment files held on disk. | A positive integer. Default: 500000000 (approximately 500 MB). | | x-queue-version | The version of the classic queue to use. | Options: 1 or 2. | | x-consumer-timeout | The duration (in milliseconds) that a consumer can remain idle before it is automatically canceled. | A positive integer that represents the number of milliseconds. For example, 60000 sets a timeout duration of one minute. | | x-single-active-consumer | When set to true, a single consumer receives messages from the queue even when multiple consumers are subscribed to it. | A boolean. | **Type**: `object` ```yaml # Examples: arguments: x-max-length: 1000 x-max-length-bytes: 4096 x-queue-type: quorum ``` ### [](#queue_declare-auto_delete)`queue_declare.auto_delete` Whether the declared queue auto-deletes when there are no active consumers. **Type**: `bool` **Default**: `false` ### [](#queue_declare-durable)`queue_declare.durable` Whether the declared queue is durable. **Type**: `bool` **Default**: `true` ### [](#queue_declare-enabled)`queue_declare.enabled` Whether to enable queue declaration. **Type**: `bool` **Default**: `false` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. This input attempts to connect to each URL in the list, in order, until a successful connection is established. It then continues to use that URL until the connection is closed. If an item in the list contains commas, it is split into multiple URLs. Requires version 3.58.0 or later. **Type**: `array` ```yaml # Examples: urls: - "amqp://guest:guest@127.0.0.1:5672/" # --- urls: - "amqp://127.0.0.1:5672/,amqp://127.0.0.2:5672/" # --- urls: - "amqp://127.0.0.1:5672/" - "amqp://127.0.0.2:5672/" ``` --- # Page 36: amqp_1 **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/amqp_1.md --- # amqp\_1 --- title: amqp_1 latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/amqp_1 page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/amqp_1.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/amqp_1.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/amqp_1/)[Output](/redpanda-connect/components/outputs/amqp_1/) **Available in:** Self-Managed Reads messages from an AMQP (1.0) server. #### Common ```yml inputs: label: "" amqp_1: urls: [] # No default (optional) source_address: "" # No default (required) ``` #### Advanced ```yml inputs: label: "" amqp_1: urls: [] # No default (optional) source_address: "" # No default (required) azure_renew_lock: false read_header: false credit: 64 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: mechanism: none user: "" password: "" ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: ```text - amqp_content_type - amqp_content_encoding - amqp_creation_time - All string typed message annotations ``` You can access these metadata fields using [function interpolation](../../../configuration/interpolation/#bloblang-queries). By setting `read_header` to `true`, additional message header properties will be added to each message: ```text - amqp_durable - amqp_priority - amqp_ttl - amqp_first_acquirer - amqp_delivery_count ``` ## [](#performance)Performance This input benefits from receiving multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages with the field `credit`. ## [](#fields)Fields ### [](#azure_renew_lock)`azure_renew_lock` Experimental: Azure service bus specific option to renew lock if processing takes more then configured lock time Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#credit)`credit` Specifies the maximum number of unacknowledged messages the sender can transmit. Once this limit is reached, no more messages will arrive until messages are acknowledged and settled. Requires version 4.26.0 or later. **Type**: `int` **Default**: `64` ### [](#read_header)`read_header` Read additional message header fields into `amqp_*` metadata properties. Requires version 4.25.0 or later. **Type**: `bool` **Default**: `false` ### [](#sasl)`sasl` Enables SASL authentication. **Type**: `object` ### [](#sasl-mechanism)`sasl.mechanism` The SASL authentication mechanism to use. **Type**: `string` **Default**: `none` | Option | Summary | | --- | --- | | anonymous | Anonymous SASL authentication. | | none | No SASL based authentication. | | plain | Plain text SASL authentication. | ### [](#sasl-password)`sasl.password` A SASL plain text password. It is recommended that you use environment variables to populate this field. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: ${PASSWORD} ``` ### [](#sasl-user)`sasl.user` A SASL plain text username. It is recommended that you use environment variables to populate this field. **Type**: `string` **Default**: `""` ```yaml # Examples: user: ${USER} ``` ### [](#source_address)`source_address` The source address to consume from. **Type**: `string` ```yaml # Examples: source_address: /foo # --- source_address: queue:/bar # --- source_address: topic:/baz ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. The first URL to successfully establish a connection will be used until the connection is closed. If an item of the list contains commas it will be expanded into multiple URLs. Requires version 4.23.0 or later. **Type**: `array` ```yaml # Examples: urls: - "amqp://guest:guest@127.0.0.1:5672/" # --- urls: - "amqp://127.0.0.1:5672/,amqp://127.0.0.2:5672/" # --- urls: - "amqp://127.0.0.1:5672/" - "amqp://127.0.0.2:5672/" ``` --- # Page 37: aws_cloudwatch_logs **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/aws_cloudwatch_logs.md --- # aws\_cloudwatch\_logs --- title: aws_cloudwatch_logs latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/aws_cloudwatch_logs page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/aws_cloudwatch_logs.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/aws_cloudwatch_logs.adoc description: Consumes log events from AWS CloudWatch Logs. page-git-created-date: "2026-03-13" page-git-modified-date: "2026-03-13" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/aws_cloudwatch_logs/ "View the Cloud version of this component"), Self-Managed Consumes log events from AWS CloudWatch Logs. Introduced in version 4.81.0. #### Common ```yml inputs: label: "" aws_cloudwatch_logs: log_group_name: "" # No default (required) log_stream_names: [] # No default (optional) log_stream_prefix: "" # No default (optional) filter_pattern: "" # No default (optional) start_time: "" # No default (optional) poll_interval: 5s auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" aws_cloudwatch_logs: log_group_name: "" # No default (required) log_stream_names: [] # No default (optional) log_stream_prefix: "" # No default (optional) filter_pattern: "" # No default (optional) start_time: "" # No default (optional) poll_interval: 5s limit: 1000 structured_log: true api_timeout: 30s auto_replay_nacks: true region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` Polls CloudWatch Log Groups for log events. Supports filtering by log streams, CloudWatch filter patterns, and configurable start times. Each log event becomes a separate message with metadata including the log group name, log stream name, timestamp, and ingestion time. > ❗ **IMPORTANT** > > This input provides at-least-once delivery. It tracks its position in memory only, so if the process restarts, it resumes from the configured `start_time` (or the beginning if not set). Duplicates can occur across restarts. For exactly-once outcomes, implement idempotent or deduplicated downstream processing. ## [](#credentials)Credentials By default, Redpanda Connect uses a shared credentials file when connecting to AWS services. You can also set credentials explicitly at the component level to transfer data across accounts. You can find out more in [AWS credentials](../../../guides/cloud/aws/). ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `cloudwatch_log_group`: The name of the log group. - `cloudwatch_log_stream`: The name of the log stream. - `cloudwatch_timestamp`: The timestamp of the log event (Unix milliseconds). - `cloudwatch_ingestion_time`: The ingestion timestamp (Unix milliseconds). - `cloudwatch_event_id`: The unique event ID. You can access these metadata fields using [function interpolation](../../../configuration/interpolation/#bloblang-queries). ## [](#requirements)Requirements Ensure that your environment meets the following requirements: - This component is only available in cgo-enabled builds of Redpanda Connect (not in the Redpanda CLI or Docker image). You can either [download a prebuilt cgo-enabled binary](../../../install/prebuilt-binary/) or [build Redpanda Connect from source with cgo enabled](../../../install/build-from-source/). ## [](#fields)Fields ### [](#api_timeout)`api_timeout` The maximum time to wait for an API request to complete. **Type**: `string` **Default**: `30s` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#credentials-2)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](../../../guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#filter_pattern)`filter_pattern` An optional CloudWatch Logs filter pattern to apply when querying log events. For syntax details, see the [CloudWatch Logs filter and pattern syntax](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/FilterAndPatternSyntax.html) documentation. **Type**: `string` ```yaml # Examples: filter_pattern: [ERROR] ``` ### [](#limit)`limit` The maximum number of log events to return in a single API call. Valid range: 1-10000. **Type**: `int` **Default**: `1000` ### [](#log_group_name)`log_group_name` The name of the CloudWatch Log Group to consume from. **Type**: `string` ```yaml # Examples: log_group_name: my-app-logs ``` ### [](#log_stream_names)`log_stream_names[]` An optional list of log stream names to consume from. If not set, events from all streams in the log group will be consumed. **Type**: `array` ```yaml # Examples: log_stream_names: - stream-1 - stream-2 ``` ### [](#log_stream_prefix)`log_stream_prefix` An optional log stream name prefix to filter streams. Only streams starting with this prefix will be consumed. **Type**: `string` ```yaml # Examples: log_stream_prefix: prod- ``` ### [](#poll_interval)`poll_interval` The interval at which to poll for new log events. **Type**: `string` **Default**: `5s` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#start_time)`start_time` The time to start consuming log events from. Can be an RFC3339 timestamp (for example, `2024-01-01T00:00:00Z`) or the string `now` to start consuming from the current time. If not set, starts from the beginning of available logs. **Type**: `string` ```yaml # Examples: start_time: 2024-01-01T00:00:00Z # --- start_time: now ``` ### [](#structured_log)`structured_log` Whether to output log events as structured JSON objects with all metadata fields, or as plain text messages with metadata stored in Redpanda Connect message metadata. **Type**: `bool` **Default**: `true` ### [](#tcp)`tcp` TCP socket configuration. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` --- # Page 38: aws_dynamodb_cdc **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/aws_dynamodb_cdc.md --- # aws\_dynamodb\_cdc --- title: aws_dynamodb_cdc latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/aws_dynamodb_cdc page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/aws_dynamodb_cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/aws_dynamodb_cdc.adoc description: Reads change data capture (CDC) events from DynamoDB Streams. page-topic-type: reference personas: data_engineer, streaming_developer, platform_operator learning-objective-1: Look up configuration options for DynamoDB CDC streaming learning-objective-2: Find metadata fields available for message processing learning-objective-3: Identify checkpointing and performance tuning settings page-git-created-date: "2026-03-02" page-git-modified-date: "2026-03-02" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/aws_dynamodb_cdc/ "View the Cloud version of this component"), Self-Managed **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Stream item-level changes from DynamoDB tables using DynamoDB Streams. This input automatically manages shards, checkpoints progress for recovery, and processes multiple shards concurrently. Introduced in version 4.79.0. Use this reference to: - Look up configuration options for DynamoDB CDC streaming - Find metadata fields available for message processing - Identify checkpointing and performance tuning settings #### Common ```yml inputs: label: "" aws_dynamodb_cdc: tables: [] checkpoint_table: redpanda_dynamodb_checkpoints start_from: trim_horizon snapshot_mode: none ``` #### Advanced ```yml inputs: label: "" aws_dynamodb_cdc: tables: [] table_discovery_mode: single table_tag_filter: "" table_discovery_interval: 5m checkpoint_table: redpanda_dynamodb_checkpoints batch_size: 1000 poll_interval: 1s start_from: trim_horizon checkpoint_limit: 1000 max_tracked_shards: 10000 throttle_backoff: 100ms snapshot_mode: none snapshot_segments: 1 snapshot_batch_size: 100 snapshot_throttle: 100ms snapshot_deduplicate: true snapshot_buffer_size: 100000 region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` ## [](#prerequisites)Prerequisites The source DynamoDB table must have [DynamoDB Streams](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html) enabled. You can enable streams with one of these view types: - `KEYS_ONLY`: Only the key attributes of the modified item - `NEW_IMAGE`: The entire item as it appears after the modification - `OLD_IMAGE`: The entire item as it appeared before the modification - `NEW_AND_OLD_IMAGES`: Both the new and old item images ## [](#checkpointing)Checkpointing Checkpoints are stored in a separate DynamoDB table (configured via `checkpoint_table`). This table is created automatically if it does not exist. On restart, the input resumes from the last checkpointed position for each shard. ## [](#alternative-components)Alternative components For better performance and longer retention (up to 1 year vs 24 hours), consider using Kinesis Data Streams for DynamoDB with the `aws_kinesis` input instead. ## [](#message-structure)Message structure Each CDC event is delivered as a JSON message with the following structure. Use these fields in your Bloblang mappings with `this.`: ```json { "eventID": "abc123-", (1) "eventName": "INSERT | MODIFY | REMOVE", (2) "eventSource": "aws:dynamodb", "awsRegion": "us-east-1", "tableName": "my-table", (3) "dynamodb": { "keys": { (4) "pk": "user#123", "sk": "profile" }, "newImage": { (5) "pk": "user#123", "sk": "profile", "name": "Alice", "email": "alice@example.com" }, "oldImage": { (6) "pk": "user#123", "sk": "profile", "name": "Alice Smith" }, "sequenceNumber": "12345678901234567890", (7) "sizeBytes": 256, "streamViewType": "NEW_AND_OLD_IMAGES" } } ``` | 1 | Unique identifier for this change event. | | --- | --- | | 2 | Type of change: INSERT (new item), MODIFY (updated item), or REMOVE (deleted item). | | 3 | Name of the source DynamoDB table. | | 4 | Primary key attributes of the changed item. Always present. | | 5 | Item state after the change. Present for INSERT and MODIFY events (requires NEW_IMAGE or NEW_AND_OLD_IMAGES stream view type). | | 6 | Item state before the change. Present for MODIFY and REMOVE events (requires OLD_IMAGE or NEW_AND_OLD_IMAGES stream view type). | | 7 | Position of this record in the shard, used for ordering and checkpointing. | > 📝 **NOTE** > > DynamoDB attribute values are automatically unmarshalled from DynamoDB’s type format (`{"S": "value"}`) to plain values (`"value"`). ### [](#example-mapping)Example mapping ```yaml pipeline: processors: - mapping: | root.event_type = this.eventName root.table = this.tableName root.keys = this.dynamodb.keys root.new_data = this.dynamodb.newImage root.old_data = this.dynamodb.oldImage ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `dynamodb_shard_id`: The shard ID from which the record was read - `dynamodb_sequence_number`: The sequence number of the record in the stream - `dynamodb_event_name`: The type of change: INSERT, MODIFY, or REMOVE - `dynamodb_table`: The name of the DynamoDB table ## [](#metrics)Metrics This input emits the following metrics: - `dynamodb_cdc_shards_tracked`: Total number of shards being tracked (gauge) - `dynamodb_cdc_shards_active`: Number of shards currently being read from (gauge) ## [](#fields)Fields ### [](#batch_size)`batch_size` Maximum number of records to read per shard in a single request. Valid range: 1-1000. **Type**: `int` **Default**: `1000` ### [](#checkpoint_limit)`checkpoint_limit` Maximum number of unacknowledged messages before forcing a checkpoint update. Lower values provide better recovery guarantees but increase write overhead. **Type**: `int` **Default**: `1000` ### [](#checkpoint_table)`checkpoint_table` DynamoDB table name for storing checkpoints. Will be created if it doesn’t exist. **Type**: `string` **Default**: `redpanda_dynamodb_checkpoints` ### [](#credentials)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](../../../guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#max_tracked_shards)`max_tracked_shards` Maximum number of shards to track simultaneously. Prevents memory issues with extremely large tables. **Type**: `int` **Default**: `10000` ### [](#poll_interval)`poll_interval` Time to wait between polling attempts when no records are available. **Type**: `string` **Default**: `1s` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#snapshot_batch_size)`snapshot_batch_size` Records per scan request during snapshot. Maximum 1000. Lower values provide better backpressure control but require more API calls. **Type**: `int` **Default**: `100` ### [](#snapshot_buffer_size)`snapshot_buffer_size` Maximum CDC events to buffer for deduplication (approximately 100 bytes per entry). If exceeded, deduplication is disabled and duplicates may be emitted. **Type**: `int` **Default**: `100000` ### [](#snapshot_deduplicate)`snapshot_deduplicate` Deduplicate records that appear in both snapshot and CDC stream. Requires buffering CDC events during snapshot. If buffer is exceeded, deduplication is disabled to prevent data loss. **Type**: `bool` **Default**: `true` ### [](#snapshot_mode)`snapshot_mode` `none`: Streams CDC events only (default). `snapshot_only`: Performs a one-time full table scan with no ongoing streaming. `snapshot_and_cdc`: Scans the entire table, then streams changes. **Type**: `string` **Default**: `none` **Options**: `none`, `snapshot_only`, `snapshot_and_cdc` ### [](#snapshot_segments)`snapshot_segments` Number of parallel scan segments (1-10). Higher parallelism scans faster but consumes more Read Capacity Units (RCUs). A lower value is safer to start with. **Type**: `int` **Default**: `1` ### [](#snapshot_throttle)`snapshot_throttle` Minimum time between scan requests per segment. Use this to limit Read Capacity Unit (RCU) consumption during snapshot. **Type**: `string` **Default**: `100ms` ### [](#start_from)`start_from` Where to start reading when no checkpoint exists. `trim_horizon` starts from the oldest available record, `latest` starts from new records. **Type**: `string` **Default**: `trim_horizon` **Options**: `trim_horizon`, `latest` ### [](#table_discovery_interval)`table_discovery_interval` Interval for rescanning and discovering new tables when using `tag` or `includelist` mode. Set to 0 to disable periodic rescanning. **Type**: `string` **Default**: `5m` ### [](#table_discovery_mode)`table_discovery_mode` `single`: Streams from tables specified in the `tables` list. `tag`: Auto-discovers tables by tags (ignores the `tables` field). `includelist`: Streams from tables in the `tables` list. Use `single` instead; `includelist` is kept for backward compatibility. **Type**: `string` **Default**: `single` **Options**: `single`, `tag`, `includelist` ### [](#table_tag_filter)`table_tag_filter` Multi-tag filter in the format `key1:v1,v2;key2:v3,v4`. Matches tables where (key1=v1 OR key1=v2) AND (key2=v3 OR key2=v4). Required when `table_discovery_mode` is `tag`. **Type**: `string` **Default**: `""` ### [](#tables)`tables[]` List of table names to stream from. For single table mode, provide one table. For multi-table mode, provide multiple tables. **Type**: `array` **Default**: `[]` ### [](#tcp)`tcp` TCP socket configuration. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#throttle_backoff)`throttle_backoff` Time to wait when applying backpressure due to too many in-flight messages. **Type**: `string` **Default**: `100ms` ## [](#examples)Examples ### [](#consume-cdc-events)Consume CDC events Read change events from a DynamoDB table with streams enabled. ```yaml input: aws_dynamodb_cdc: tables: [my-table] region: us-east-1 ``` ### [](#start-from-latest)Start from latest Only process new changes, ignoring existing stream data. ```yaml input: aws_dynamodb_cdc: tables: [orders] start_from: latest region: us-west-2 ``` ### [](#snapshot-and-cdc)Snapshot and CDC Scan all existing records, then stream ongoing changes. ```yaml input: aws_dynamodb_cdc: tables: [products] snapshot_mode: snapshot_and_cdc snapshot_segments: 5 region: us-east-1 ``` ### [](#auto-discover-tables-by-tag)Auto-discover tables by tag Automatically discover and stream from all tables with a specific tag. ```yaml input: aws_dynamodb_cdc: table_discovery_mode: tag table_tag_filter: "stream-enabled:true" table_discovery_interval: 5m region: us-east-1 ``` ### [](#auto-discover-tables-by-multiple-tags)Auto-discover tables by multiple tags Discover tables matching multiple tag criteria with OR logic per key, AND logic across keys. ```yaml input: aws_dynamodb_cdc: table_discovery_mode: tag table_tag_filter: "environment:prod,staging;team:data,analytics" table_discovery_interval: 5m region: us-east-1 # Matches tables with: (environment=prod OR environment=staging) AND (team=data OR team=analytics) ``` ### [](#stream-from-multiple-specific-tables)Stream from multiple specific tables Stream from an explicit list of tables simultaneously. ```yaml input: aws_dynamodb_cdc: table_discovery_mode: includelist tables: - orders - customers - products region: us-west-2 ``` ## [](#suggested-reading)Suggested reading For common patterns including filtering events, routing to Kafka or S3, and detecting changed fields, see the [DynamoDB CDC Patterns](../../../cookbooks/dynamodb_cdc/) cookbook. --- # Page 39: aws_kinesis **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/aws_kinesis.md --- # aws\_kinesis --- title: aws_kinesis latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/aws_kinesis page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/aws_kinesis.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/aws_kinesis.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/aws_kinesis/)[Output](/redpanda-connect/components/outputs/aws_kinesis/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/aws_kinesis/ "View the Cloud version of this component"), Self-Managed Receive messages from one or more Kinesis streams. Introduced in version 3.36.0. #### Common ```yml inputs: label: "" aws_kinesis: streams: [] # No default (required) dynamodb: table: "" create: false billing_mode: PAY_PER_REQUEST read_capacity_units: 0 write_capacity_units: 0 region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) checkpoint_limit: 1024 auto_replay_nacks: true commit_period: 5s steal_grace_period: 2s start_from_oldest: true batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml inputs: label: "" aws_kinesis: streams: [] # No default (required) dynamodb: table: "" create: false billing_mode: PAY_PER_REQUEST read_capacity_units: 0 write_capacity_units: 0 region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) checkpoint_limit: 1024 auto_replay_nacks: true commit_period: 5s steal_grace_period: 2s rebalance_period: 30s lease_period: 30s start_from_oldest: true region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` Consumes messages from one or more Kinesis streams either by automatically balancing shards across other instances of this input, or by consuming shards listed explicitly. The latest message sequence consumed by this input is stored within a [DynamoDB table](#table-schema), which allows it to resume at the correct sequence of the shard during restarts. This table is also used for coordination across distributed inputs when shard balancing. Redpanda Connect will not store a consumed sequence unless it is acknowledged at the output level, which ensures at-least-once delivery guarantees. ## [](#ordering)Ordering By default messages of a shard can be processed in parallel, up to a limit determined by the field `checkpoint_limit`. However, if strict ordered processing is required then this value must be set to 1 in order to process shard messages in lock-step. When doing so it is recommended that you perform batching at this component for performance as it will not be possible to batch lock-stepped messages at the output level. ## [](#table-schema)Table schema It’s possible to configure Redpanda Connect to create the DynamoDB table required for coordination if it does not already exist. However, if you wish to create this yourself (recommended) then create a table with a string HASH key `StreamID` and a string RANGE key `ShardID`. ## [](#batching)Batching Use the `batching` fields to configure an optional [batching policy](../../../configuration/batching/#batch-policy). Each stream shard will be batched separately in order to ensure that acknowledgements aren’t contaminated. ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#batching-2)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#checkpoint_limit)`checkpoint_limit` The maximum gap between the in flight sequence versus the latest acknowledged sequence at a given time. Increasing this limit enables parallel processing and batching at the output level to work on individual shards. Any given sequence will not be committed unless all messages under that offset are delivered in order to preserve at least once delivery guarantees. **Type**: `int` **Default**: `1024` ### [](#commit_period)`commit_period` The period of time between each update to the checkpoint table. **Type**: `string` **Default**: `5s` ### [](#credentials)`credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services](../../../guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` The role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the AWS credentials in use. This is a required value for short-term credentials. **Type**: `string` ### [](#dynamodb)`dynamodb` Determines the table used for storing and accessing the latest consumed sequence for shards, and for coordinating balanced consumers of streams. **Type**: `object` ### [](#dynamodb-billing_mode)`dynamodb.billing_mode` When creating the table determines the billing mode. **Type**: `string` **Default**: `PAY_PER_REQUEST` **Options**: `PROVISIONED`, `PAY_PER_REQUEST` ### [](#dynamodb-create)`dynamodb.create` Whether, if the table does not exist, it should be created. **Type**: `bool` **Default**: `false` ### [](#dynamodb-credentials)`dynamodb.credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services](../../../guides/cloud/aws/). **Type**: `object` ### [](#dynamodb-credentials-from_ec2_role)`dynamodb.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#dynamodb-credentials-id)`dynamodb.credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#dynamodb-credentials-profile)`dynamodb.credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#dynamodb-credentials-role)`dynamodb.credentials.role` The role ARN to assume. **Type**: `string` ### [](#dynamodb-credentials-role_external_id)`dynamodb.credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#dynamodb-credentials-secret)`dynamodb.credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#dynamodb-credentials-token)`dynamodb.credentials.token` The token for the AWS credentials in use. This is a required value for short-term credentials. **Type**: `string` ### [](#dynamodb-endpoint)`dynamodb.endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#dynamodb-read_capacity_units)`dynamodb.read_capacity_units` Set the provisioned read capacity when creating the table with a `billing_mode` of `PROVISIONED`. **Type**: `int` **Default**: `0` ### [](#dynamodb-region)`dynamodb.region` The AWS region to target. **Type**: `string` ### [](#dynamodb-table)`dynamodb.table` The name of the table to access. **Type**: `string` **Default**: `""` ### [](#dynamodb-tcp)`dynamodb.tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#dynamodb-tcp-connect_timeout)`dynamodb.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#dynamodb-tcp-keep_alive)`dynamodb.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#dynamodb-tcp-keep_alive-count)`dynamodb.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#dynamodb-tcp-keep_alive-idle)`dynamodb.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#dynamodb-tcp-keep_alive-interval)`dynamodb.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#dynamodb-tcp-tcp_user_timeout)`dynamodb.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#dynamodb-write_capacity_units)`dynamodb.write_capacity_units` Set the provisioned write capacity when creating the table with a `billing_mode` of `PROVISIONED`. **Type**: `int` **Default**: `0` ### [](#endpoint)`endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#lease_period)`lease_period` The period of time after which a client that has failed to update a shard checkpoint is assumed to be inactive. **Type**: `string` **Default**: `30s` ### [](#rebalance_period)`rebalance_period` The period of time between each attempt to rebalance shards across clients. **Type**: `string` **Default**: `30s` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#start_from_oldest)`start_from_oldest` Whether to consume from the oldest message when a sequence does not yet exist for the stream. **Type**: `bool` **Default**: `true` ### [](#steal_grace_period)`steal_grace_period` Determines how long beyond the next commit period a client will wait when stealing a shard for the current owner to store a checkpoint. A longer value increases the time taken to balance shards but reduces the likelihood of processing duplicate messages. **Type**: `string` **Default**: `2s` ### [](#streams)`streams[]` One or more Kinesis data streams to consume from. Streams can either be specified by their name or full ARN. Shards of a stream are automatically balanced across consumers by coordinating through the provided DynamoDB table. Multiple comma separated streams can be listed in a single element. Shards are automatically distributed across consumers of a stream by coordinating through the provided DynamoDB table. Alternatively, it’s possible to specify an explicit shard to consume from with a colon after the stream name, e.g. `foo:0` would consume the shard `0` of the stream `foo`. **Type**: `array` ```yaml # Examples: streams: - foo - "arn:aws:kinesis:*:111122223333:stream/my-stream" ``` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` --- # Page 40: aws_s3 **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/aws_s3.md --- # aws\_s3 --- title: aws_s3 latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/aws_s3 page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/aws_s3.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/aws_s3.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-03-19" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/aws_s3/)[Cache](/redpanda-connect/components/caches/aws_s3/)[Output](/redpanda-connect/components/outputs/aws_s3/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/aws_s3/ "View the Cloud version of this component"), Self-Managed Downloads objects within an Amazon S3 bucket, optionally filtered by a prefix, either by walking the items in the bucket or by streaming upload notifications in real time. #### Common ```yml inputs: label: "" aws_s3: bucket: "" prefix: "" scanner: to_the_end: {} sqs: url: "" endpoint: "" key_path: Records.*.s3.object.key bucket_path: Records.*.s3.bucket.name envelope_path: "" delay_period: "" max_messages: 10 wait_time_seconds: 0 nack_visibility_timeout: 0 ``` #### Advanced ```yml inputs: label: "" aws_s3: bucket: "" prefix: "" region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) force_path_style_urls: false delete_objects: false scanner: to_the_end: {} sqs: url: "" endpoint: "" key_path: Records.*.s3.object.key bucket_path: Records.*.s3.bucket.name envelope_path: "" delay_period: "" max_messages: 10 wait_time_seconds: 0 nack_visibility_timeout: 0 ``` ## [](#stream-objects-on-upload-with-sqs)Stream objects on upload with SQS A common pattern for consuming S3 objects is to emit upload notification events from the bucket either directly to an SQS queue, or to an SNS topic that is consumed by an SQS queue, and then have your consumer listen for events that prompt it to download the newly uploaded objects. More information about this pattern and how to set it up can be found in the [Amazon S3 docs](https://docs.aws.amazon.com/AmazonS3/latest/dev/ways-to-add-notification-config-to-bucket.html). Redpanda Connect is able to follow this pattern when you configure an `sqs.url`, where it consumes events from SQS and downloads only the object keys contained in those events. For this to work, Redpanda Connect needs to know where within the event the key and bucket names can be found, specified as [dot paths](../../../configuration/field_paths/) with the fields `sqs.key_path` and `sqs.bucket_path`. The default values for these fields should already be correct when following the guide above. If your notification events are being routed to SQS via an SNS topic, the events are enveloped by SNS, in which case you also need to specify the field `sqs.envelope_path`, which in the case of SNS to SQS will usually be `Message`. When using SQS, make sure you have sensible values for `sqs.max_messages` and also the visibility timeout of the queue itself. When Redpanda Connect consumes an S3 object the SQS message that triggered it is not deleted until the S3 object has been sent onwards. This ensures at-least-once crash resiliency, but also means that if the S3 object takes longer to process than the visibility timeout of your queue, then the same objects might be processed multiple times. ## [](#download-large-files)Download large files When downloading large files, process them in streamed parts to avoid loading the entire file into memory at once. To do this, specify a [`scanner`](#scanner) that determines how to break the input into smaller individual messages. ## [](#bucket-and-prefix)Bucket and prefix The `bucket` field accepts a bucket name only, not an ARN. For example, use `my-bucket`, not `arn:aws:s3:::my-bucket`. The `prefix` field accepts a single string. To consume from multiple prefixes in the same bucket, use multiple `aws_s3` inputs in a [`broker` input](../broker/): ```yaml input: broker: inputs: - aws_s3: bucket: my-bucket prefix: logs/app1/ - aws_s3: bucket: my-bucket prefix: logs/app2/ ``` ## [](#credentials)Credentials By default, Redpanda Connect uses a shared credentials file when connecting to AWS services. You can also set credentials explicitly at the component level to transfer data across accounts. You can find out more in [AWS credentials](../../../guides/cloud/aws/). ## [](#s3-compatible-storage)S3-compatible storage The `endpoint` and `force_path_style_urls` fields let you connect to S3-compatible storage services such as Cloudflare R2, MinIO, or DigitalOcean Spaces. For Cloudflare R2, set `endpoint` to your account endpoint URL and enable `force_path_style_urls`: ```yaml input: aws_s3: bucket: r2-bucket endpoint: https://.r2.cloudflarestorage.com force_path_style_urls: true region: auto credentials: id: secret: ``` Find your account ID in the Cloudflare dashboard under **R2 > Overview > Account Details**. Generate API credentials under **R2 > Manage R2 API Tokens**. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - s3\_key - s3\_bucket - s3\_last\_modified\_unix - s3\_last\_modified (RFC3339) - s3\_content\_type - s3\_content\_encoding - s3\_version\_id - All user defined metadata You can access these metadata fields using [function interpolation](../../../configuration/interpolation/#bloblang-queries). User-defined metadata is case insensitive in AWS, so keys are often received in capitalized form. To normalize them, map all metadata keys to lowercase or uppercase using a Bloblang mapping such as `meta = meta().map_each_key(key → key.lowercase())`. ## [](#fields)Fields ### [](#bucket)`bucket` The bucket to consume from. If the field `sqs.url` is specified this field is optional. **Type**: `string` **Default**: `""` ### [](#credentials-2)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](../../../guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#delete_objects)`delete_objects` Whether to delete downloaded objects from the bucket once they are processed. **Type**: `bool` **Default**: `false` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#force_path_style_urls)`force_path_style_urls` Forces the client API to use path style URLs for downloading keys, which is often required when connecting to custom endpoints. **Type**: `bool` **Default**: `false` ### [](#prefix)`prefix` An optional path prefix, if set only objects with the prefix are consumed when walking a bucket. **Type**: `string` **Default**: `""` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#scanner)`scanner` The [scanner](../../scanners/about/) by which the stream of bytes consumed will be broken out into individual messages. Scanners are useful for processing large sources of data without holding the entirety of it within memory. For example, the `csv` scanner allows you to process individual CSV rows without loading the entire CSV file in memory at once. Requires version 4.25.0 or later. **Type**: `scanner` **Default**: ```yaml to_the_end: {} ``` ### [](#sqs)`sqs` Consume SQS messages in order to trigger key downloads. **Type**: `object` ### [](#sqs-bucket_path)`sqs.bucket_path` A [dot path](../../../configuration/field_paths/) whereby the bucket name can be found in SQS messages. **Type**: `string` **Default**: `Records.*.s3.bucket.name` ### [](#sqs-delay_period)`sqs.delay_period` An optional period of time to wait from when a notification was originally sent to when the target key download is attempted. **Type**: `string` **Default**: `""` ```yaml # Examples: delay_period: 10s # --- delay_period: 5m ``` ### [](#sqs-endpoint)`sqs.endpoint` A custom endpoint to use when connecting to SQS. **Type**: `string` **Default**: `""` ### [](#sqs-envelope_path)`sqs.envelope_path` A [dot path](../../../configuration/field_paths/) of a field to extract an enveloped JSON payload for further extracting the key and bucket from SQS messages. This is specifically useful when subscribing an SQS queue to an SNS topic that receives bucket events. **Type**: `string` **Default**: `""` ```yaml # Examples: envelope_path: Message ``` ### [](#sqs-key_path)`sqs.key_path` A [dot path](../../../configuration/field_paths/) whereby object keys are found in SQS messages. **Type**: `string` **Default**: `Records.*.s3.object.key` ### [](#sqs-max_messages)`sqs.max_messages` The maximum number of SQS messages to consume from each request. **Type**: `int` **Default**: `10` ### [](#sqs-nack_visibility_timeout)`sqs.nack_visibility_timeout` Custom SQS Nack Visibility timeout in seconds. Default is 0 **Type**: `int` **Default**: `0` ### [](#sqs-url)`sqs.url` An optional SQS URL to connect to. When specified this queue will control which objects are downloaded. **Type**: `string` **Default**: `""` ### [](#sqs-wait_time_seconds)`sqs.wait_time_seconds` Whether to set the wait time. Enabling this activates long-polling. Valid values: 0 to 20. **Type**: `int` **Default**: `0` ### [](#tcp)`tcp` TCP socket configuration. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` --- # Page 41: aws_sqs **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/aws_sqs.md --- # aws\_sqs --- title: aws_sqs latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/aws_sqs page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/aws_sqs.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/aws_sqs.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/aws_sqs/)[Output](/redpanda-connect/components/outputs/aws_sqs/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/aws_sqs/ "View the Cloud version of this component"), Self-Managed Consume messages from an AWS SQS URL. #### Common ```yml inputs: label: "" aws_sqs: url: "" # No default (required) max_outstanding_messages: 1000 ``` #### Advanced ```yml inputs: label: "" aws_sqs: url: "" # No default (required) delete_message: true reset_visibility: true max_number_of_messages: 10 max_outstanding_messages: 1000 wait_time_seconds: 0 message_timeout: 30s region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` ## [](#credentials)Credentials By default, Redpanda Connect uses a shared credentials file when connecting to AWS services. You can also set credentials explicitly at the component level, which allows you to transfer data across accounts. To find out more, see [Amazon Web Services](../../../guides/cloud/aws/). ## [](#metadata)Metadata This input adds the following metadata fields to each message: - sqs\_message\_id - sqs\_receipt\_handle - sqs\_approximate\_receive\_count - All message attributes You can access these metadata fields using [function interpolation](../../../configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#credentials-2)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](../../../guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#delete_message)`delete_message` Whether to delete the consumed message when it’s acknowledged. Set to `false` to handle the deletion using a different mechanism. **Type**: `bool` **Default**: `true` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#max_number_of_messages)`max_number_of_messages` The maximum number of messages that Redpanda Connect can return each time it polls the SQS URL. Enter values from `1` to `10` only. **Type**: `int` **Default**: `10` ### [](#max_outstanding_messages)`max_outstanding_messages` The maximum number of pending messages that Redpanda Connect can have in flight at the same time. **Type**: `int` **Default**: `1000` ### [](#message_timeout)`message_timeout` The maximum time allowed to process a received message before Redpanda Connect refreshes the [receipt handle](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-queue-message-identifiers.html), and the message becomes visible in the queue again. Redpanda Connect attempts to refresh the receipt handle after half of the timeout has elapsed. **Type**: `string` **Default**: `30s` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#reset_visibility)`reset_visibility` Whether to set the visibility timeout of the consumed message to zero if Redpanda Connect receives a negative acknowledgement. Set to `false` to use the [queue’s visibility timeout](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html) for each message rather than releasing the message immediately for reprocessing. Requires version 3.58.0 or later. **Type**: `bool` **Default**: `true` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#url)`url` The SQS URL to consume from. **Type**: `string` ### [](#wait_time_seconds)`wait_time_seconds` Whether to set a wait time (in seconds). Enter values from `1` to `20` to enable wait times and to activate [log polling](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-short-and-long-polling.html) for queued messages. **Type**: `int` **Default**: `0` --- # Page 42: azure_blob_storage **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/azure_blob_storage.md --- # azure\_blob\_storage --- title: azure_blob_storage latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/azure_blob_storage page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/azure_blob_storage.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/azure_blob_storage.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/azure_blob_storage/)[Output](/redpanda-connect/components/outputs/azure_blob_storage/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/azure_blob_storage/ "View the Cloud version of this component"), Self-Managed Downloads objects within an Azure Blob Storage container, optionally filtered by a prefix. Introduced in version 3.36.0. #### Common ```yml inputs: label: "" azure_blob_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" container: "" # No default (required) prefix: "" scanner: to_the_end: {} targets_input: "" # No default (optional) ``` #### Advanced ```yml inputs: label: "" azure_blob_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" container: "" # No default (required) prefix: "" scanner: to_the_end: {} delete_objects: false targets_input: "" # No default (optional) ``` Supports multiple authentication methods but only one of the following is required: - `storage_connection_string` - `storage_account` and `storage_access_key` - `storage_account` and `storage_sas_token` - `storage_account` to access via [DefaultAzureCredential](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#DefaultAzureCredential) If multiple are set then the `storage_connection_string` is given priority. If the `storage_connection_string` does not contain the `AccountName` parameter, please specify it in the `storage_account` field. ## [](#download-large-files)Download large files When downloading large files it’s often necessary to process it in streamed parts in order to avoid loading the entire file in memory at a given time. In order to do this a [`scanner`](#scanner) can be specified that determines how to break the input into smaller individual messages. ## [](#stream-new-files)Stream new files By default this input will consume all files found within the target container and will then gracefully terminate. This is referred to as a "batch" mode of operation. However, it’s possible to instead configure a container as [an Event Grid source](https://learn.microsoft.com/en-gb/azure/event-grid/event-schema-blob-storage) and then use this as a [`targets_input`](#targets_input), in which case new files are consumed as they’re uploaded and Redpanda Connect will continue listening for and downloading files as they arrive. This is referred to as a "streamed" mode of operation. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - blob\_storage\_key - blob\_storage\_container - blob\_storage\_last\_modified - blob\_storage\_last\_modified\_unix - blob\_storage\_content\_type - blob\_storage\_content\_encoding - All user defined metadata You can access these metadata fields using [function interpolation](../../../configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#container)`container` The name of the container from which to download blobs. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#delete_objects)`delete_objects` Whether to delete downloaded objects from the blob once they are processed. **Type**: `bool` **Default**: `false` ### [](#prefix)`prefix` An optional path prefix, if set only objects with the prefix are consumed. **Type**: `string` **Default**: `""` ### [](#scanner)`scanner` The [scanner](../../scanners/about/) by which the stream of bytes consumed will be broken out into individual messages. Scanners are useful for processing large sources of data without holding the entirety of it within memory. For example, the `csv` scanner allows you to process individual CSV rows without loading the entire CSV file in memory at once. Requires version 4.25.0 or later. **Type**: `scanner` **Default**: ```yaml to_the_end: {} ``` ### [](#storage_access_key)`storage_access_key` The storage account access key. This field is ignored if `storage_connection_string` is set. **Type**: `string` **Default**: `""` ### [](#storage_account)`storage_account` The storage account to access. This field is ignored if `storage_connection_string` is set. **Type**: `string` **Default**: `""` ### [](#storage_connection_string)`storage_connection_string` A storage account connection string. This field is required if `storage_account` and `storage_access_key` / `storage_sas_token` are not set. **Type**: `string` **Default**: `""` ### [](#storage_sas_token)`storage_sas_token` The storage account SAS token. This field is ignored if `storage_connection_string` or `storage_access_key` are set. **Type**: `string` **Default**: `""` ### [](#targets_input)`targets_input` > ⚠️ **CAUTION** > > This is an experimental field that provides an optional source of download targets, configured as a [regular Redpanda Connect input](../about/). Each message yielded by this input should be a single structured object containing a field `name`, which represents the blob to be downloaded. This requires setting up [Azure Blob Storage as an Event Grid source](https://learn.microsoft.com/en-gb/azure/event-grid/event-schema-blob-storage) and an associated event handler that a Redpanda Connect input can read from. For example, use either one of the following: - [Azure Event Hubs](https://learn.microsoft.com/en-gb/azure/event-grid/handler-event-hubs) using the `kafka` input - [Namespace topics](https://learn.microsoft.com/en-gb/azure/event-grid/handler-event-grid-namespace-topic) using the `mqtt` input Requires version 4.27.0 or later. **Type**: `input` ```yaml # Examples: targets_input: mqtt: topics: - some-topic urls: - example.westeurope-1.ts.eventgrid.azure.net:8883 processors: - unarchive: format: json_array - mapping: |- if this.eventType == "Microsoft.Storage.BlobCreated" { root.name = this.data.url.parse_url().path.trim_prefix("/foocontainer/") } else { root = deleted() } ``` --- # Page 43: azure_cosmosdb **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/azure_cosmosdb.md --- # azure\_cosmosdb --- title: azure_cosmosdb latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/azure_cosmosdb page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/azure_cosmosdb.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/azure_cosmosdb.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/azure_cosmosdb/)[Output](/redpanda-connect/components/outputs/azure_cosmosdb/)[Processor](/redpanda-connect/components/processors/azure_cosmosdb/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/azure_cosmosdb/ "View the Cloud version of this component"), Self-Managed Executes a SQL query against [Azure CosmosDB](https://learn.microsoft.com/en-us/azure/cosmos-db/introduction) and creates a batch of messages from each page of items. Introduced in version v4.25.0. #### Common ```yml inputs: label: "" azure_cosmosdb: endpoint: "" # No default (optional) account_key: "" # No default (optional) connection_string: "" # No default (optional) database: "" # No default (required) container: "" # No default (required) partition_keys_map: "" # No default (required) query: "" # No default (required) args_mapping: "" # No default (optional) auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" azure_cosmosdb: endpoint: "" # No default (optional) account_key: "" # No default (optional) connection_string: "" # No default (optional) database: "" # No default (required) container: "" # No default (required) partition_keys_map: "" # No default (required) query: "" # No default (required) args_mapping: "" # No default (optional) batch_count: -1 auto_replay_nacks: true ``` ## [](#cross-partition-queries)Cross-partition queries Cross-partition queries are currently not supported by the underlying driver. For every query, the PartitionKey values must be known in advance and specified in the config. [See details](https://github.com/Azure/azure-sdk-for-go/issues/18578#issuecomment-1222510989). ## [](#credentials)Credentials You can use one of the following authentication mechanisms: - Set the `endpoint` field and the `account_key` field - Set only the `endpoint` field to use [DefaultAzureCredential](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#DefaultAzureCredential) - Set the `connection_string` field ## [](#metadata)Metadata This component adds the following metadata fields to each message: ```none - activity_id - request_charge ``` You can access these metadata fields using [function interpolation](../../../configuration/interpolation/#bloblang-queries). ## [](#examples)Examples ### [](#query-container)Query container Execute a parametrized SQL query to select documents from a container. ```yaml input: azure_cosmosdb: endpoint: http://localhost:8080 account_key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw== database: blobbase container: blobfish partition_keys_map: root = "AbyssalPlain" query: SELECT * FROM blobfish AS b WHERE b.species = @species args_mapping: | root = [ { "Name": "@species", "Value": "smooth-head" }, ] ``` ## [](#fields)Fields ### [](#account_key)`account_key` Account key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ```yaml # Examples: account_key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw== ``` ### [](#args_mapping)`args_mapping` A [Bloblang mapping](../../../guides/bloblang/about/) that, for each message, creates a list of arguments to use with the query. **Type**: `string` ```yaml # Examples: args_mapping: |- root = [ { "Name": "@name", "Value": "benthos" }, ] ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#batch_count)`batch_count` The maximum number of messages that should be accumulated into each batch. Use '-1' specify dynamic page size. **Type**: `int` **Default**: `-1` ### [](#connection_string)`connection_string` Connection string. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ```yaml # Examples: connection_string: AccountEndpoint=https://localhost:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==; ``` ### [](#container)`container` Container. **Type**: `string` ```yaml # Examples: container: testcontainer ``` ### [](#database)`database` Database. **Type**: `string` ```yaml # Examples: database: testdb ``` ### [](#endpoint)`endpoint` CosmosDB endpoint. **Type**: `string` ```yaml # Examples: endpoint: https://localhost:8081 ``` ### [](#partition_keys_map)`partition_keys_map` A [Bloblang mapping](../../../guides/bloblang/about/) which should evaluate to a single partition key value or an array of partition key values of type string, integer or boolean. Currently, hierarchical partition keys are not supported so only one value may be provided. **Type**: `string` ```yaml # Examples: partition_keys_map: root = "blobfish" # --- partition_keys_map: root = 41 # --- partition_keys_map: root = true # --- partition_keys_map: root = null # --- partition_keys_map: root = now().ts_format("2006-01-02") ``` ### [](#query)`query` The query to execute **Type**: `string` ```yaml # Examples: query: SELECT c.foo FROM testcontainer AS c WHERE c.bar = "baz" AND c.timestamp < @timestamp ``` ## [](#cosmosdb-emulator)CosmosDB emulator If you wish to run the CosmosDB emulator that is referenced in the documentation [here](https://learn.microsoft.com/en-us/azure/cosmos-db/linux-emulator), the following Docker command should do the trick: ```bash > docker run --rm -it -p 8081:8081 --name=cosmosdb -e AZURE_COSMOS_EMULATOR_PARTITION_COUNT=10 -e AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE=false mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator ``` Note: `AZURE_COSMOS_EMULATOR_PARTITION_COUNT` controls the number of partitions that will be supported by the emulator. The bigger the value, the longer it takes for the container to start up. Additionally, instead of installing the container self-signed certificate which is exposed via `[https://localhost:8081/_explorer/emulator.pem](https://localhost:8081/_explorer/emulator.pem)`, you can run [mitmproxy](https://mitmproxy.org/) like so: ```bash > mitmproxy -k --mode "reverse:https://localhost:8081" ``` Then you can access the CosmosDB UI via `[http://localhost:8080/_explorer/index.html](http://localhost:8080/_explorer/index.html)` and use `[http://localhost:8080](http://localhost:8080)` as the CosmosDB endpoint. --- # Page 44: azure_queue_storage **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/azure_queue_storage.md --- # azure\_queue\_storage --- title: azure_queue_storage latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/azure_queue_storage page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/azure_queue_storage.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/azure_queue_storage.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/azure_queue_storage/)[Output](/redpanda-connect/components/outputs/azure_queue_storage/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/azure_queue_storage/ "View the Cloud version of this component"), Self-Managed Dequeue objects from an Azure Storage Queue. Introduced in version 3.42.0. #### Common ```yml inputs: label: "" azure_queue_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" queue_name: "" # No default (required) ``` #### Advanced ```yml inputs: label: "" azure_queue_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" queue_name: "" # No default (required) dequeue_visibility_timeout: 30s max_in_flight: 10 track_properties: false ``` This input adds the following metadata fields to each message: ```none - queue_storage_insertion_time - queue_storage_queue_name - queue_storage_message_lag (if 'track_properties' set to true) - All user defined queue metadata ``` Only one authentication method is required, `storage_connection_string` or `storage_account` and `storage_access_key`. If both are set then the `storage_connection_string` is given priority. ## [](#fields)Fields ### [](#dequeue_visibility_timeout)`dequeue_visibility_timeout` The timeout duration until a dequeued message gets visible again, 30s by default Requires version 3.45.0 or later. **Type**: `string` **Default**: `30s` ### [](#max_in_flight)`max_in_flight` The maximum number of unprocessed messages to fetch at a given time. **Type**: `int` **Default**: `10` ### [](#queue_name)`queue_name` The name of the source storage queue. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: queue_name: foo_queue # --- queue_name: ${! env("MESSAGE_TYPE").lowercase() } ``` ### [](#storage_access_key)`storage_access_key` The storage account access key. This field is ignored if `storage_connection_string` is set. **Type**: `string` **Default**: `""` ### [](#storage_account)`storage_account` The storage account to access. This field is ignored if `storage_connection_string` is set. **Type**: `string` **Default**: `""` ### [](#storage_connection_string)`storage_connection_string` A storage account connection string. This field is required if `storage_account` and `storage_access_key` / `storage_sas_token` are not set. **Type**: `string` **Default**: `""` ### [](#track_properties)`track_properties` If set to `true` the queue is polled on each read request for information such as the queue message lag. These properties are added to consumed messages as metadata, but will also have a negative performance impact. **Type**: `bool` **Default**: `false` --- # Page 45: azure_table_storage **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/azure_table_storage.md --- # azure\_table\_storage --- title: azure_table_storage latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/azure_table_storage page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/azure_table_storage.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/azure_table_storage.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/azure_table_storage/)[Output](/redpanda-connect/components/outputs/azure_table_storage/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/azure_table_storage/ "View the Cloud version of this component"), Self-Managed Queries an Azure Storage Account Table, optionally with multiple filters. Introduced in version 4.10.0. #### Common ```yml inputs: label: "" azure_table_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" table_name: "" # No default (required) ``` #### Advanced ```yml inputs: label: "" azure_table_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" table_name: "" # No default (required) filter: "" select: "" page_size: 1000 ``` Queries an Azure Storage Account Table, optionally with multiple filters. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - table\_storage\_name - row\_num You can access these metadata fields using [function interpolation](../../../configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#filter)`filter` OData filter expression. Is not set all rows are returned. Valid operators are `eq, ne, gt, lt, ge and le` **Type**: `string` **Default**: `""` ```yaml # Examples: filter: PartitionKey eq 'foo' and RowKey gt '1000' ``` ### [](#page_size)`page_size` Maximum number of records to return on each page. **Type**: `int` **Default**: `1000` ### [](#select)`select` Select expression using OData notation. Limits the columns on each record to just those requested. **Type**: `string` **Default**: `""` ```yaml # Examples: select: PartitionKey,RowKey,Foo,Bar,Timestamp ``` ### [](#storage_access_key)`storage_access_key` The storage account access key. This field is ignored if `storage_connection_string` is set. **Type**: `string` **Default**: `""` ### [](#storage_account)`storage_account` The storage account to access. This field is ignored if `storage_connection_string` is set. **Type**: `string` **Default**: `""` ### [](#storage_connection_string)`storage_connection_string` A storage account connection string. This field is required if `storage_account` and `storage_access_key` / `storage_sas_token` are not set. **Type**: `string` **Default**: `""` ### [](#storage_sas_token)`storage_sas_token` The storage account SAS token. This field is ignored if `storage_connection_string` or `storage_access_key` are set. **Type**: `string` **Default**: `""` ### [](#table_name)`table_name` The table to read messages from. **Type**: `string` ```yaml # Examples: table_name: Foo ``` --- # Page 46: batched **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/batched.md --- # batched --- title: batched latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/batched page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/batched.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/batched.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/batched/ "View the Cloud version of this component"), Self-Managed Consumes data from a child input and applies a batching policy to the stream. Introduced in version 4.11.0. #### Common ```yml inputs: label: "" batched: child: "" # No default (required) policy: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml inputs: label: "" batched: child: "" # No default (required) policy: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` Batching at the input level is sometimes useful for processing across micro-batches, and can also sometimes be a useful performance trick. However, most inputs are fine without it so unless you have a specific plan for batching this component is not worth using. ## [](#fields)Fields ### [](#child)`child` The child input. **Type**: `input` ### [](#policy)`policy` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: policy: byte_size: 5000 count: 0 period: 1s # --- policy: count: 10 period: 1s # --- policy: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#policy-byte_size)`policy.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#policy-check)`policy.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#policy-count)`policy.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#policy-period)`policy.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#policy-processors)`policy.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` --- # Page 47: beanstalkd **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/beanstalkd.md --- # beanstalkd --- title: beanstalkd latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/beanstalkd page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/beanstalkd.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/beanstalkd.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/beanstalkd/)[Output](/redpanda-connect/components/outputs/beanstalkd/) **Available in:** Self-Managed Reads messages from a Beanstalkd queue. Introduced in version 4.7.0. #### Common ```yml inputs: label: "" beanstalkd: address: "" # No default (required) ``` #### Advanced ```yml inputs: label: "" beanstalkd: address: "" # No default (required) ``` ## [](#fields)Fields ### [](#address)`address` An address to connect to. **Type**: `string` ```yaml # Examples: address: 127.0.0.1:11300 ``` --- # Page 48: broker **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/broker.md --- # broker --- title: broker latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/broker page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/broker.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/broker.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/broker/)[Output](/redpanda-connect/components/outputs/broker/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/broker/ "View the Cloud version of this component"), Self-Managed Allows you to combine multiple inputs into a single stream of data, where each input will be read in parallel. #### Common ```yml inputs: label: "" broker: inputs: [] # No default (required) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml inputs: label: "" broker: copies: 1 inputs: [] # No default (required) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` A broker type is configured with its own list of input configurations and a field to specify how many copies of the list of inputs should be created. Adding more input types allows you to combine streams from multiple sources into one. For example, reading from both RabbitMQ and Kafka: ```yaml input: broker: copies: 1 inputs: - amqp_0_9: urls: - amqp://guest:guest@localhost:5672/ consumer_tag: benthos-consumer queue: benthos-queue # Optional list of input specific processing steps processors: - mapping: | root.message = this root.meta.link_count = this.links.length() root.user.age = this.user.age.number() - kafka: addresses: - localhost:9092 client_id: benthos_kafka_input consumer_group: benthos_consumer_group topics: [ benthos_stream:0 ] ``` If the number of copies is greater than zero the list will be copied that number of times. For example, if your inputs were of type foo and bar, with 'copies' set to '2', you would end up with two 'foo' inputs and two 'bar' inputs. ## [](#batching)Batching It’s possible to configure a [batch policy](../../../configuration/batching/#batch-policy) with a broker using the `batching` fields. When doing this the feeds from all child inputs are combined. Some inputs do not support broker based batching and specify this in their documentation. ## [](#processors)Processors It is possible to configure [processors](../../processors/about/) at the broker level, where they will be applied to _all_ child inputs, as well as on the individual child inputs. If you have processors at both the broker level _and_ on child inputs then the broker processors will be applied _after_ the child nodes processors. ## [](#fields)Fields ### [](#batching-2)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#copies)`copies` Whatever is specified within `inputs` will be created this many times. **Type**: `int` **Default**: `1` ### [](#inputs)`inputs[]` A list of inputs to create. **Type**: `input` --- # Page 49: cassandra **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/cassandra.md --- # cassandra --- title: cassandra latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/cassandra page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/cassandra.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/cassandra.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/cassandra/)[Output](/redpanda-connect/components/outputs/cassandra/) **Available in:** Self-Managed Executes a find query and creates a message for each row received. #### Common ```yml inputs: label: "" cassandra: addresses: [] # No default (required) timeout: 600ms reconnect_interval: 60s query: "" # No default (required) auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" cassandra: addresses: [] # No default (required) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] password_authenticator: enabled: false username: "" password: "" disable_initial_host_lookup: false max_retries: 3 backoff: initial_interval: 1s max_interval: 5s timeout: 600ms host_selection_policy: local_dc: "" # No default (optional) local_rack: "" # No default (optional) reconnect_interval: 60s exponential_reconnection: max_retries: "" # No default (required) initial_interval: "" # No default (required) max_interval: "" # No default (required) query: "" # No default (required) auto_replay_nacks: true ``` ## [](#examples)Examples ### [](#minimal-select-cassandrascylla)Minimal Select (Cassandra/Scylla) Let’s presume that we have 3 Cassandra nodes, like in this tutorial by Sebastian Sigl from freeCodeCamp: [https://www.freecodecamp.org/news/the-apache-cassandra-beginner-tutorial/](https://www.freecodecamp.org/news/the-apache-cassandra-beginner-tutorial/) Then if we want to select everything from the table users\_by\_country, we should use the configuration below. If we specify the stdin output, the result will look like: ```json {"age":23,"country":"UK","first_name":"Bob","last_name":"Sandler","user_email":"bob@email.com"} ``` This configuration also works for Scylla. ```yaml input: cassandra: addresses: - 172.17.0.2 query: 'SELECT * FROM learn_cassandra.users_by_country' ``` ## [](#fields)Fields ### [](#addresses)`addresses[]` A list of Cassandra nodes to connect to. Multiple comma separated addresses can be specified on a single line. **Type**: `array` ```yaml # Examples: addresses: - "localhost:9042" # --- addresses: - "foo:9042" - "bar:9042" # --- addresses: - "foo:9042,bar:9042" ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#backoff)`backoff` Control time intervals between retry attempts. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` The initial period to wait between retry attempts. The retry interval increases for each failed attempt, up to the `backoff.max_interval` value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `1s` ### [](#backoff-max_interval)`backoff.max_interval` The maximum period to wait between retry attempts. **Type**: `string` **Default**: `5s` ### [](#disable_initial_host_lookup)`disable_initial_host_lookup` If enabled the driver will not attempt to get host info from the system.peers table. This can speed up queries but will mean that data\_centre, rack and token information will not be available. **Type**: `bool` **Default**: `false` ### [](#exponential_reconnection)`exponential_reconnection` Configure exponential backoff for reconnection attempts to DOWN nodes. When enabled, this replaces the driver’s default constant reconnection policy with an exponential backoff strategy that gradually increases the delay between reconnection attempts. This reduces connection storm scenarios during widespread outages while ensuring eventual recovery. Requires version 4.66.0 or later. **Type**: `object` ### [](#exponential_reconnection-initial_interval)`exponential_reconnection.initial_interval` The initial period to wait between retry attempts. **Type**: `string` ### [](#exponential_reconnection-max_interval)`exponential_reconnection.max_interval` The maximum period to wait between retry attempts. **Type**: `string` ### [](#exponential_reconnection-max_retries)`exponential_reconnection.max_retries` The maximum number of retry attempts. **Type**: `int` ### [](#host_selection_policy)`host_selection_policy` Advanced host selection policy settings for Cassandra clusters. Use these options to optimize query routing in multi-datacenter (DC) and multi-rack deployments. By specifying a local DC and rack, you can ensure queries are directed to the closest nodes, reducing latency and improving fault tolerance. If not set, the default policy is round-robin across all available nodes. Host selection is always token-aware if the token can be calculated from query. Requires version 4.61.0 or later. **Type**: `object` ```yaml # Examples: host_selection_policy: local_dc: dc-east local_rack: rack1 ``` ### [](#host_selection_policy-local_dc)`host_selection_policy.local_dc` The name of the local datacenter to prioritize for query routing. Enables DC-aware host selection, ensuring queries are sent to nodes within this datacenter whenever possible. Recommended for clusters spanning multiple datacenters to minimize cross-DC traffic. **Type**: `string` ### [](#host_selection_policy-local_rack)`host_selection_policy.local_rack` The name of the local rack to prioritize for query routing. Requires `local_dc` to be set. Enables rack-aware host selection, further optimizing query placement within the specified datacenter. Useful for deployments with multiple racks per datacenter to improve resilience and reduce intra-DC latency. **Type**: `string` ### [](#max_retries)`max_retries` The maximum number of retries before giving up on a request. **Type**: `int` **Default**: `3` ### [](#password_authenticator)`password_authenticator` Optional configuration of Cassandra authentication parameters. **Type**: `object` ### [](#password_authenticator-enabled)`password_authenticator.enabled` Whether to use password authentication **Type**: `bool` **Default**: `false` ### [](#password_authenticator-password)`password_authenticator.password` The password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#password_authenticator-username)`password_authenticator.username` The username to authenticate as. **Type**: `string` **Default**: `""` ### [](#query)`query` A query to execute. **Type**: `string` ### [](#reconnect_interval)`reconnect_interval` The interval at which Redpanda Connect attempts to reconnect to Cassandra nodes that are marked as DOWN. This setting helps maintain connectivity in unstable network conditions or during node maintenance. Use Go duration format such as `30s`, `1m`, or `5m`. Setting this too low may create unnecessary connection attempts, while setting it too high may delay recovery from network issues. Requires version 4.66.0 or later. **Type**: `string` **Default**: `60s` ### [](#timeout)`timeout` The client connection timeout. **Type**: `string` **Default**: `600ms` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` --- # Page 50: cockroachdb_changefeed **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/cockroachdb_changefeed.md --- # cockroachdb\_changefeed --- title: cockroachdb_changefeed latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/cockroachdb_changefeed page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/cockroachdb_changefeed.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/cockroachdb_changefeed.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Available in:** Self-Managed Listens to a [CockroachDB Core Changefeed](https://www.cockroachlabs.com/docs/stable/changefeed-examples) and creates a message for each row received. Each message is a json object looking like: ```json { "primary_key": "[\"1a7ff641-3e3b-47ee-94fe-a0cadb56cd8f\", 2]", // stringifed JSON array "row": "{\"after\": {\"k\": \"1a7ff641-3e3b-47ee-94fe-a0cadb56cd8f\", \"v\": 2}, \"updated\": \"1637953249519902405.0000000000\"}", // stringified JSON object "table": "strm_2" } ``` #### Common ```yml inputs: label: "" cockroachdb_changefeed: dsn: "" # No default (required) tables: [] # No default (required) cursor_cache: "" # No default (optional) auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" cockroachdb_changefeed: dsn: "" # No default (required) tls: skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tables: [] # No default (required) cursor_cache: "" # No default (optional) options: [] # No default (optional) auto_replay_nacks: true ``` This input will continue to listen to the changefeed until shutdown. A backfill of the full current state of the table will be delivered upon each run unless a cache is configured for storing cursor timestamps, as this is how Redpanda Connect keeps track as to which changes have been successfully delivered. Note: You must have `SET CLUSTER SETTING kv.rangefeed.enabled = true;` on your CRDB cluster, for more information refer to [the official CockroachDB documentation](https://www.cockroachlabs.com/docs/stable/changefeed-examples?filters=core). ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#cursor_cache)`cursor_cache` A [cache resource](https://www.docs.redpanda.com/redpanda-connect/components/caches/about) to use for storing the current latest cursor that has been successfully delivered, this allows Redpanda Connect to continue from that cursor upon restart, rather than consume the entire state of the table. **Type**: `string` ### [](#dsn)`dsn` A Data Source Name to identify the target database. **Type**: `string` ```yaml # Examples: dsn: postgres://user:password@example.com:26257/defaultdb?sslmode=require ``` ### [](#options)`options[]` A list of options to be included in the changefeed (WITH X, Y…​). > 📝 **NOTE** > > Both the CURSOR option and UPDATED will be ignored from these options when a `cursor_cache` is specified, as they are set explicitly by Redpanda Connect in this case. **Type**: `array` ```yaml # Examples: options: - "virtual_columns=\"omitted\"" ``` ### [](#tables)`tables[]` CSV of tables to be included in the changefeed **Type**: `array` ```yaml # Examples: tables: - table1 - table2 ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` --- # Page 51: csv **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/csv.md --- # csv --- title: csv latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/csv page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/csv.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/csv.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/csv/)[Scanner](/redpanda-connect/components/scanners/csv/) **Available in:** Self-Managed Reads one or more CSV files as structured records following the format described in RFC 4180. #### Common ```yml inputs: label: "" csv: paths: [] # No default (required) parse_header_row: true delimiter: , lazy_quotes: false auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" csv: paths: [] # No default (required) parse_header_row: true delimiter: , lazy_quotes: false delete_on_finish: false batch_count: 1 auto_replay_nacks: true ``` This input offers more control over CSV parsing than the [`file` input](../file/). When parsing with a header row each line of the file will be consumed as a structured object, where the key names are determined from the header now. For example, the following CSV file: ```csv foo,bar,baz first foo,first bar,first baz second foo,second bar,second baz ``` Would produce the following messages: ```json {"foo":"first foo","bar":"first bar","baz":"first baz"} {"foo":"second foo","bar":"second bar","baz":"second baz"} ``` If, however, the field `parse_header_row` is set to `false` then arrays are produced instead, like follows: ```json ["first foo","first bar","first baz"] ["second foo","second bar","second baz"] ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: ```text - header - path - mod_time_unix - mod_time (RFC3339) ``` You can access these metadata fields using [function interpolation](../../../configuration/interpolation/#bloblang-queries). Note: The `header` field is only set when `parse_header_row` is `true`. ### [](#output-csv-column-order)Output CSV column order When [creating CSV](../../../guides/bloblang/advanced/#creating-csv) from Redpanda Connect messages, the columns must be sorted lexicographically to make the output deterministic. Alternatively, when using the `csv` input, one can leverage the `header` metadata field to retrieve the column order: ```yaml input: csv: paths: - ./foo.csv - ./bar.csv parse_header_row: true processors: - mapping: | map escape_csv { root = if this.re_match("[\"\n,]+") { "\"" + this.replace_all("\"", "\"\"") + "\"" } else { this } } let header = if count(@path) == 1 { @header.map_each(c -> c.apply("escape_csv")).join(",") + "\n" } else { "" } root = $header + @header.map_each(c -> this.get(c).string().apply("escape_csv")).join(",") output: file: path: ./output/${! @path.filepath_split().index(-1) } ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#batch_count)`batch_count` Optionally process records in batches. This can help to speed up the consumption of exceptionally large CSV files. When the end of the file is reached the remaining records are processed as a (potentially smaller) batch. **Type**: `int` **Default**: `1` ### [](#delete_on_finish)`delete_on_finish` Whether to delete input files from the disk once they are fully consumed. **Type**: `bool` **Default**: `false` ### [](#delimiter)`delimiter` The delimiter to use for splitting values in each record. It must be a single character. **Type**: `string` **Default**: `,` ### [](#lazy_quotes)`lazy_quotes` If set to `true`, a quote may appear in an unquoted field and a non-doubled quote may appear in a quoted field. Requires version 4.1.0 or later. **Type**: `bool` **Default**: `false` ### [](#parse_header_row)`parse_header_row` Whether to reference the first row as a header row. If set to true the output structure for messages will be an object where field keys are determined by the header row. Otherwise, each message will consist of an array of values from the corresponding CSV row. **Type**: `bool` **Default**: `true` ### [](#paths)`paths[]` A list of file paths to read from. Each file will be read sequentially until the list is exhausted, at which point the input will close. Glob patterns are supported, including super globs (double star). **Type**: `array` ```yaml # Examples: paths: - /tmp/foo.csv - /tmp/bar/*.csv - /tmp/data/**/*.csv ``` --- # Page 52: discord **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/discord.md --- # discord --- title: discord latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/discord page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/discord.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/discord.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/discord/)[Output](/redpanda-connect/components/outputs/discord/) **Available in:** Self-Managed Consumes messages posted in a Discord channel. #### Common ```yml inputs: label: "" discord: channel_id: "" # No default (required) bot_token: "" # No default (required) cache: "" # No default (required) auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" discord: channel_id: "" # No default (required) bot_token: "" # No default (required) cache: "" # No default (required) cache_key: last_message_id auto_replay_nacks: true ``` This input works by authenticating as a bot using token based authentication. The ID of the newest message consumed and acked is stored in a cache in order to perform a backfill of unread messages each time the input is initialised. Ideally this cache should be persisted across restarts. ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#bot_token)`bot_token` A bot token used for authentication. **Type**: `string` ### [](#cache)`cache` A cache resource to use for performing unread message backfills, the ID of the last message received will be stored in this cache and used for subsequent requests. **Type**: `string` ### [](#cache_key)`cache_key` The key identifier used when storing the ID of the last message received. **Type**: `string` **Default**: `last_message_id` ### [](#channel_id)`channel_id` A discord channel ID to consume messages from. **Type**: `string` --- # Page 53: dynamic **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/dynamic.md --- # dynamic --- title: dynamic latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/dynamic page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/dynamic.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/dynamic.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/dynamic/)[Output](/redpanda-connect/components/outputs/dynamic/) **Available in:** Self-Managed A special broker type where the inputs are identified by unique labels and can be created, changed and removed during runtime via a REST HTTP interface. ```yml inputs: label: "" dynamic: inputs: {} prefix: "" ``` ## [](#fields)Fields ### [](#inputs)`inputs` A map of inputs to statically create. **Type**: `input` **Default**: `{}` ### [](#prefix)`prefix` A path prefix for HTTP endpoints that are registered. **Type**: `string` **Default**: `""` ## [](#endpoints)Endpoints ### [](#get-inputs)GET `/inputs` Returns a JSON object detailing all dynamic inputs, providing information such as their current uptime and configuration. ### [](#get-inputsid)GET `/inputs/{id}` Returns the configuration of an input. ### [](#post-inputsid)POST `/inputs/{id}` Creates or updates an input with a configuration provided in the request body (in YAML or JSON format). ### [](#delete-inputsid)DELETE `/inputs/{id}` Stops and removes an input. ### [](#get-inputsiduptime)GET `/inputs/{id}/uptime` Returns the uptime of an input as a duration string (of the form "72h3m0.5s"), or "stopped" in the case where the input has gracefully terminated. --- # Page 54: file **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/file.md --- # file --- title: file latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/file page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/file.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/file.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/file/)[Cache](/redpanda-connect/components/caches/file/)[Output](/redpanda-connect/components/outputs/file/) **Available in:** Self-Managed Consumes data from files on disk, emitting messages according to a chosen codec. #### Common ```yml inputs: label: "" file: paths: [] # No default (required) scanner: lines: {} auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" file: paths: [] # No default (required) scanner: lines: {} delete_on_finish: false auto_replay_nacks: true ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: ```text - path - mod_time_unix - mod_time (RFC3339) ``` You can access these metadata fields using [function interpolation](../../../configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#delete_on_finish)`delete_on_finish` Whether to delete input files from the disk once they are fully consumed. **Type**: `bool` **Default**: `false` ### [](#paths)`paths[]` A list of paths to consume sequentially. Glob patterns are supported, including super globs (double star). **Type**: `array` ### [](#scanner)`scanner` The [scanner](../../scanners/about/) by which the stream of bytes consumed will be broken out into individual messages. Scanners are useful for processing large sources of data without holding the entirety of it within memory. For example, the `csv` scanner allows you to process individual CSV rows without loading the entire CSV file in memory at once. Requires version 4.25.0 or later. **Type**: `scanner` **Default**: ```yaml lines: {} ``` ## [](#examples)Examples ### [](#read-a-bunch-of-csvs)Read a Bunch of CSVs If we wished to consume a directory of CSV files as structured documents we can use a glob pattern and the `csv` scanner: ```yaml input: file: paths: [ ./data/*.csv ] scanner: csv: {} ``` --- # Page 55: gcp_bigquery_select **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/gcp_bigquery_select.md --- # gcp\_bigquery\_select --- title: gcp_bigquery_select latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/gcp_bigquery_select page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/gcp_bigquery_select.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/gcp_bigquery_select.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/gcp_bigquery_select/)[Processor](/redpanda-connect/components/processors/gcp_bigquery_select/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/gcp_bigquery_select/ "View the Cloud version of this component"), Self-Managed Executes a `SELECT` query against BigQuery and creates a message for each row received. Introduced in version 3.63.0. ```yml inputs: label: "" gcp_bigquery_select: project: "" # No default (required) credentials_json: "" table: "" # No default (required) columns: [] # No default (required) where: "" # No default (optional) auto_replay_nacks: true job_labels: {} priority: "" args_mapping: "" # No default (optional) prefix: "" # No default (optional) suffix: "" # No default (optional) ``` Once the rows from the query are exhausted, this input shuts down, allowing the pipeline to gracefully terminate (or the next input in a [sequence](../sequence/) to execute). ## [](#examples)Examples ### [](#word-counts)Word counts Here we query the public corpus of Shakespeare’s works to generate a stream of the top 10 words that are 3 or more characters long: ```yaml input: gcp_bigquery_select: project: sample-project table: bigquery-public-data.samples.shakespeare columns: - word - sum(word_count) as total_count where: length(word) >= ? suffix: | GROUP BY word ORDER BY total_count DESC LIMIT 10 args_mapping: | root = [ 3 ] ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` An optional [Bloblang mapping](../../../guides/bloblang/about/) which should evaluate to an array of values matching in size to the number of placeholder arguments in the field `where`. **Type**: `string` ```yaml # Examples: args_mapping: root = [ "article", now().ts_format("2006-01-02") ] ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#columns)`columns[]` A list of columns to query. **Type**: `array` ### [](#credentials_json)`credentials_json` Base64-encoded Google Service Account credentials in JSON format (optional). Use this field to authenticate with Google Cloud services. For more information about creating service account credentials, see [Google’s service account documentation](https://developers.google.com/workspace/guides/create-credentials#create_credentials_for_a_service_account). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#job_labels)`job_labels` A list of labels to add to the query job. **Type**: `string` **Default**: `{}` ### [](#prefix)`prefix` An optional prefix to prepend to the select query (before SELECT). **Type**: `string` ### [](#priority)`priority` The priority with which to schedule the query. **Type**: `string` **Default**: `""` ### [](#project)`project` GCP project where the query job will execute. **Type**: `string` ### [](#suffix)`suffix` An optional suffix to append to the select query. **Type**: `string` ### [](#table)`table` Fully-qualified BigQuery table name to query. **Type**: `string` ```yaml # Examples: table: bigquery-public-data.samples.shakespeare ``` ### [](#where)`where` An optional where clause to add. Placeholder arguments are populated with the `args_mapping` field. Placeholders should always be question marks (`?`). **Type**: `string` ```yaml # Examples: where: type = ? and created_at > ? # --- where: user_id = ? ``` --- # Page 56: gcp_cloud_storage **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/gcp_cloud_storage.md --- # gcp\_cloud\_storage --- title: gcp_cloud_storage latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/gcp_cloud_storage page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/gcp_cloud_storage.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/gcp_cloud_storage.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/gcp_cloud_storage/)[Cache](/redpanda-connect/components/caches/gcp_cloud_storage/)[Output](/redpanda-connect/components/outputs/gcp_cloud_storage/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/gcp_cloud_storage/ "View the Cloud version of this component"), Self-Managed Downloads objects within a Google Cloud Storage bucket, optionally filtered by a prefix. Introduced in version 3.43.0. #### Common ```yml inputs: label: "" gcp_cloud_storage: bucket: "" # No default (required) prefix: "" credentials_json: "" scanner: to_the_end: {} ``` #### Advanced ```yml inputs: label: "" gcp_cloud_storage: bucket: "" # No default (required) prefix: "" credentials_json: "" scanner: to_the_end: {} delete_objects: false ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: ```none - gcs_key - gcs_bucket - gcs_last_modified - gcs_last_modified_unix - gcs_content_type - gcs_content_encoding - All user defined metadata ``` You can access these metadata fields using [function interpolation](../../../configuration/interpolation/#bloblang-queries). ### [](#credentials)Credentials By default Redpanda Connect will use a shared credentials file when connecting to GCP services. You can find out more in [Google Cloud Platform](../../../guides/cloud/gcp/). ## [](#fields)Fields ### [](#bucket)`bucket` The name of the bucket from which to download objects. **Type**: `string` ### [](#credentials_json)`credentials_json` Base64-encoded Google Service Account credentials in JSON format (optional). Use this field to authenticate with Google Cloud services. For more information about creating service account credentials, see [Google’s service account documentation](https://developers.google.com/workspace/guides/create-credentials#create_credentials_for_a_service_account). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#delete_objects)`delete_objects` Whether to delete downloaded objects from the bucket once they are processed. **Type**: `bool` **Default**: `false` ### [](#prefix)`prefix` Optional path prefix, if set only objects with the prefix are consumed. **Type**: `string` **Default**: `""` ### [](#scanner)`scanner` The [scanner](../../scanners/about/) by which the stream of bytes consumed will be broken out into individual messages. Scanners are useful for processing large sources of data without holding the entirety of it within memory. For example, the `csv` scanner allows you to process individual CSV rows without loading the entire CSV file in memory at once. Requires version 4.25.0 or later. **Type**: `scanner` **Default**: ```yaml to_the_end: {} ``` --- # Page 57: gcp_pubsub **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/gcp_pubsub.md --- # gcp\_pubsub --- title: gcp_pubsub latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/gcp_pubsub page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/gcp_pubsub.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/gcp_pubsub.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/gcp_pubsub/)[Output](/redpanda-connect/components/outputs/gcp_pubsub/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/gcp_pubsub/ "View the Cloud version of this component"), Self-Managed Consumes messages from a GCP Cloud Pub/Sub subscription. #### Common ```yml inputs: label: "" gcp_pubsub: project: "" # No default (required) credentials_json: "" subscription: "" # No default (required) endpoint: "" sync: false max_outstanding_messages: 1000 max_outstanding_bytes: 1000000000 ``` #### Advanced ```yml inputs: label: "" gcp_pubsub: project: "" # No default (required) credentials_json: "" subscription: "" # No default (required) endpoint: "" sync: false max_outstanding_messages: 1000 max_outstanding_bytes: 1000000000 create_subscription: enabled: false topic: "" ``` For information on how to set up credentials see [this guide](https://cloud.google.com/docs/authentication/production). ## [](#metadata)Metadata This input adds the following metadata fields to each message: - gcp\_pubsub\_publish\_time\_unix - The time at which the message was published to the topic. - gcp\_pubsub\_delivery\_attempt - When dead lettering is enabled, this is set to the number of times PubSub has attempted to deliver a message. - All message attributes You can access these metadata fields using [function interpolation](../../../configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#create_subscription)`create_subscription` Allows you to configure the input subscription and creates if it doesn’t exist. **Type**: `object` ### [](#create_subscription-enabled)`create_subscription.enabled` Whether to configure subscription or not. **Type**: `bool` **Default**: `false` ### [](#create_subscription-topic)`create_subscription.topic` Defines the topic that the subscription should be vinculated to. **Type**: `string` **Default**: `""` ### [](#credentials_json)`credentials_json` Base64-encoded Google Service Account credentials in JSON format (optional). Use this field to authenticate with Google Cloud services. For more information about creating service account credentials, see [Google’s service account documentation](https://developers.google.com/workspace/guides/create-credentials#create_credentials_for_a_service_account). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#endpoint)`endpoint` An optional endpoint to override the default of `pubsub.googleapis.com:443`. This can be used to connect to a region specific pubsub endpoint. For a list of valid values, see [this document](https://cloud.google.com/pubsub/docs/reference/service_apis_overview#list_of_regional_endpoints). **Type**: `string` **Default**: `""` ```yaml # Examples: endpoint: us-central1-pubsub.googleapis.com:443 # --- endpoint: us-west3-pubsub.googleapis.com:443 ``` ### [](#max_outstanding_bytes)`max_outstanding_bytes` The maximum number of outstanding pending messages to be consumed measured in bytes. **Type**: `int` **Default**: `1000000000` ### [](#max_outstanding_messages)`max_outstanding_messages` The maximum number of outstanding pending messages to be consumed at a given time. **Type**: `int` **Default**: `1000` ### [](#project)`project` The project ID of the target subscription. **Type**: `string` ### [](#subscription)`subscription` The target subscription ID. **Type**: `string` ### [](#sync)`sync` Enable synchronous pull mode. **Type**: `bool` **Default**: `false` --- # Page 58: gcp_spanner_cdc **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/gcp_spanner_cdc.md --- # gcp\_spanner\_cdc --- title: gcp_spanner_cdc latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/gcp_spanner_cdc page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/gcp_spanner_cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/gcp_spanner_cdc.adoc description: Creates an input that consumes from a spanner change stream. page-git-created-date: "2025-06-25" page-git-modified-date: "2026-01-13" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/gcp_spanner_cdc/ "View the Cloud version of this component"), Self-Managed **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Creates an input that consumes from a spanner change stream. Introduced in version 4.56.0. #### Common ```yaml inputs: label: "" gcp_spanner_cdc: credentials_json: "" project_id: "" # No default (required) instance_id: "" # No default (required) database_id: "" # No default (required) stream_id: "" # No default (required) start_timestamp: "" end_timestamp: "" batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) auto_replay_nacks: true ``` #### Advanced ```yaml inputs: label: "" gcp_spanner_cdc: credentials_json: "" project_id: "" # No default (required) instance_id: "" # No default (required) database_id: "" # No default (required) stream_id: "" # No default (required) start_timestamp: "" end_timestamp: "" heartbeat_interval: 10s metadata_table: "" min_watermark_cache_ttl: 5s allowed_mod_types: [] # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) auto_replay_nacks: true ``` Consumes change records from a Google Cloud Spanner change stream. This input allows you to track and process database changes in real-time, making it useful for data replication, event-driven architectures, and maintaining derived data stores. The input reads from a specified change stream within a Spanner database and converts each change record into a message. The message payload contains the change records in JSON format, and metadata is added with details about the Spanner instance, database, and stream. Change streams provide a way to track mutations to your Spanner database tables. For more information about Spanner change streams, refer to the [Google Cloud documentation](https://cloud.google.com/spanner/docs/change-streams). ## [](#fields)Fields ### [](#allowed_mod_types)`allowed_mod_types[]` List of modification types to process. If not specified, all modification types are processed. Allowed values: INSERT, UPDATE, DELETE **Type**: `array` ```yaml # Examples: allowed_mod_types: - INSERT - UPDATE - DELETE ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay messages that are rejected (nacked) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The maximum total size (in bytes) that a batch can reach before it is passed on for processing or delivery (flushed). When the combined size of all messages in the batch exceeds this limit, the batch is immediately sent to the next stage (such as a processor or output). Set to `0` to disable size-based batching. When disabled, messages are flushed based on other conditions (such as `batching.count` or `batching.period`). **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages at which the batch should be flushed. Set the value to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The length of time after which an incomplete batch should be flushed regardless of its size. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. For example, `1s` flushes a batch after one second. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, so any attempt to split it into smaller batches with these processors will be ignored. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#credentials_json)`credentials_json` Base64-encoded JSON credentials file for authenticating to GCP with a service account. If not provided, Application Default Credentials (ADC) is used. For more information about how to create a service account and obtain the credentials JSON, see the [Google Cloud documentation](https://cloud.google.com/docs/authentication/getting-started). **Type**: `string` **Default**: `""` ### [](#database_id)`database_id` The ID of the Spanner database to read from. This is the name of the database as it appears in the Spanner console or API. For more information about how to create a Spanner database, see the [Google Cloud documentation](https://cloud.google.com/spanner/docs/create-manage-databases). **Type**: `string` ### [](#end_timestamp)`end_timestamp` The timestamp at which to stop reading change records from the change stream. This is an optional field that allows you to limit the range of change records processed by the input. The timestamp should be in RFC3339 format, such as `2023-10-01T00:00:00Z`. If not provided, the input reads all available change records up to the current time. **Type**: `string` **Default**: `""` ```yaml # Examples: end_timestamp: 2022-01-01T00:00:00Z ``` ### [](#heartbeat_interval)`heartbeat_interval` The interval at which to send heartbeat messages to the output. Heartbeat messages are sent to indicate that the input is still active and processing changes. This can help prevent timeouts in downstream systems. Supported time units are `ns`, `us`, `ms`, `s`, `m`, and `h`. For example, `1s` sends a heartbeat every second. **Type**: `string` **Default**: `10s` ### [](#instance_id)`instance_id` The ID of the Spanner instance to read from. This is the name of the instance as it appears in the Spanner console or API. For more information about how to create a Spanner instance, see the [Google Cloud documentation](https://cloud.google.com/spanner/docs/create-manage-instances). **Type**: `string` ### [](#metadata_table)`metadata_table` The table to store metadata in (default: `cdc_metadata_`). **Type**: `string` **Default**: `""` ### [](#min_watermark_cache_ttl)`min_watermark_cache_ttl` Sets how frequently to query Spanner for the minimum watermark. **Type**: `string` **Default**: `5s` ### [](#project_id)`project_id` The ID of the GCP project that contains the Spanner instance and database. This is the name of the project as it appears in the GCP console or API. For more information about how to create a GCP project, see the [Google Cloud documentation](https://cloud.google.com/resource-manager/docs/creating-managing-projects). **Type**: `string` ### [](#start_timestamp)`start_timestamp` The timestamp at which to start reading change records from the change stream. This is an optional field that allows you to limit the range of change records processed by the input. The timestamp should be in RFC3339 format, such as `2023-10-01T00:00:00Z` (default: current time). **Type**: `string` **Default**: `""` ```yaml # Examples: start_timestamp: 2022-01-01T00:00:00Z ``` ### [](#stream_id)`stream_id` The name of the change stream to track. The stream must exist in the Spanner database. To create a change stream, follow the [Google Cloud documentation](https://cloud.google.com/spanner/docs/change-streams/manage). **Type**: `string` --- # Page 59: generate **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/generate.md --- # generate --- title: generate latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/generate page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/generate.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/generate.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/generate/ "View the Cloud version of this component"), Self-Managed Generates messages at a given interval using a [Bloblang](../../../guides/bloblang/about/) mapping executed without a context. This allows you to generate messages for testing your pipeline configs. Introduced in version 3.40.0. #### Common ```yml inputs: label: "" generate: mapping: "" # No default (required) interval: 1s count: 0 batch_size: 1 auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" generate: mapping: "" # No default (required) interval: 1s count: 0 batch_size: 1 auto_replay_nacks: true ``` ## [](#examples)Examples ### [](#cron-scheduled-processing)Cron Scheduled Processing A common use case for the generate input is to trigger processors on a schedule so that the processors themselves can behave similarly to an input. The following configuration reads rows from a PostgreSQL table every 5 minutes. ```yaml input: generate: interval: '@every 5m' mapping: 'root = {}' processors: - sql_select: driver: postgres dsn: postgres://foouser:foopass@localhost:5432/testdb?sslmode=disable table: foo columns: [ "*" ] ``` ### [](#generate-100-rows)Generate 100 Rows The generate input can be used as a convenient way to generate test data. The following example generates 100 rows of structured data by setting an explicit count. The interval field is set to empty, which means data is generated as fast as the downstream components can consume it. ```yaml input: generate: count: 100 interval: "" mapping: | root = if random_int() % 2 == 0 { { "type": "foo", "foo": "is yummy" } } else { { "type": "bar", "bar": "is gross" } } ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#batch_size)`batch_size` The number of generated messages that should be accumulated into each batch flushed at the specified interval. **Type**: `int` **Default**: `1` ### [](#count)`count` An optional number of messages to generate, if set above 0 the specified number of messages is generated and then the input will shut down. **Type**: `int` **Default**: `0` ### [](#interval)`interval` The time interval at which messages should be generated, expressed either as a duration string or as a cron expression. If set to an empty string messages will be generated as fast as downstream services can process them. Cron expressions can specify a timezone by prefixing the expression with `TZ=`, where the location name corresponds to a file within the IANA Time Zone database. **Type**: `string` **Default**: `1s` ```yaml # Examples: interval: 5s # --- interval: 1m # --- interval: 1h # --- interval: @every 1s # --- interval: 0,30 */2 * * * * # --- interval: TZ=Europe/London 30 3-6,20-23 * * * ``` ### [](#mapping)`mapping` A [Bloblang](../../../guides/bloblang/about/) mapping to use for generating messages. **Type**: `string` ```yaml # Examples: mapping: root = "hello world" # --- mapping: root = {"test":"message","id":uuid_v4()} ``` --- # Page 60: git **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/git.md --- # git --- title: git latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/git page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/git.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/git.adoc page-git-created-date: "2025-05-02" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/git/ "View the Cloud version of this component"), Self-Managed Clones a Git repository, reads its contents, then polls for new commits at a configurable interval. Any updates are emitted as new messages. Introduced in version 4.51.0. ```yml inputs: label: "" git: repository_url: "" # No default (required) branch: main poll_interval: 10s include_patterns: [] exclude_patterns: [] max_file_size: 10485760 checkpoint_cache: "" # No default (optional) checkpoint_key: git_last_commit auth: basic: username: "" password: "" ssh_key: private_key_path: "" private_key: "" passphrase: "" token: value: "" auto_replay_nacks: true ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `git_file_path` - `git_file_size` - `git_file_mode` - `git_file_modified` - `git_commit` - `git_mime_type` - `git_is_binary` - `git_deleted` (when a source file is deleted) You can access these metadata fields using [function interpolation](../../../configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#auth)`auth` Options for authenticating with your Git repository. **Type**: `object` ### [](#auth-basic)`auth.basic` Allows you to specify basic authentication. **Type**: `object` ### [](#auth-basic-password)`auth.basic.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#auth-basic-username)`auth.basic.username` The username to use for authentication. **Type**: `string` **Default**: `""` ### [](#auth-ssh_key)`auth.ssh_key` Allows you to specify SSH key authentication. **Type**: `object` ### [](#auth-ssh_key-passphrase)`auth.ssh_key.passphrase` The passphrase for your SSH private key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#auth-ssh_key-private_key)`auth.ssh_key.private_key` Your private SSH key. When using encrypted keys, you must also set a value for [`private_key_passphrase`](#auth-ssh_key-passphrase). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#auth-ssh_key-private_key_path)`auth.ssh_key.private_key_path` The path to your private SSH key file. When using encrypted keys, you must also set a value for [`private_key_passphrase`](#auth-ssh_key-passphrase). **Type**: `string` **Default**: `""` ### [](#auth-token)`auth.token` Allows you to specify token-based authentication. **Type**: `object` ### [](#auth-token-value)`auth.token.value` The token value to use for token-based authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay messages that are rejected (nacked) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#branch)`branch` The repository branch to check out. **Type**: `string` **Default**: `main` ### [](#checkpoint_cache)`checkpoint_cache` Specify a [`cache`](../../caches/about/) resource to store the last processed commit hash. After a restart, Redpanda Connect can then continue processing changes from where it left off, avoiding the need to reprocess all detected updates. **Type**: `string` ### [](#checkpoint_key)`checkpoint_key` The key to use when storing the last processed commit hash in the cache. **Type**: `string` **Default**: `git_last_commit` ### [](#exclude_patterns)`exclude_patterns[]` A list of file patterns to exclude. For example, you could choose not to read content from certain Git directories or image files: `'.git/**', '**/*.png'`. These patterns take precedence over `include_patterns`. The following patterns are supported: - Glob patterns: **, `/`**`*/`, `?` - Character ranges: `[a-z]`. Escape any character with a special meaning using a backslash. **Type**: `array` **Default**: `[]` ### [](#include_patterns)`include_patterns[]` A list of file patterns to read from. For example, you could read content from only Markdown and YAML files: `'***/**.md', 'configs/*.yaml'`. The following patterns are supported: - Glob patterns: **, `/`**`*/`, `?` - Character ranges: `[a-z]`. Escape any character with a special meaning using a backslash. If this field is left empty, all files are read from. **Type**: `array` **Default**: `[]` ### [](#max_file_size)`max_file_size` The maximum size of files to read from (in bytes). Files that exceed this limit are skipped. Set to `0` for unlimited file sizes. **Type**: `int` **Default**: `10485760` ### [](#poll_interval)`poll_interval` How frequently this input polls the Git repository for changes. **Type**: `string` **Default**: `10s` ```yaml # Examples: poll_interval: 10s ``` ### [](#repository_url)`repository_url` The URL of the Git repository to clone. **Type**: `string` ```yaml # Examples: repository_url: https://github.com/username/repo.git ``` --- # Page 61: hdfs **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/hdfs.md --- # hdfs --- title: hdfs latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/hdfs page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/hdfs.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/hdfs.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/hdfs/)[Output](/redpanda-connect/components/outputs/hdfs/) **Available in:** Self-Managed Reads files from a HDFS directory, where each discrete file will be consumed as a single message payload. ```yml inputs: label: "" hdfs: hosts: [] # No default (required) user: "" directory: "" # No default (required) ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: - hdfs\_name - hdfs\_path You can access these metadata fields using [function interpolation](../../../configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#directory)`directory` The directory to consume from. **Type**: `string` ### [](#hosts)`hosts[]` A list of target host addresses to connect to. **Type**: `array` ```yaml # Examples: hosts: localhost:9000 ``` ### [](#user)`user` A user ID to connect as. **Type**: `string` **Default**: `""` --- # Page 62: http_client **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/http_client.md --- # http\_client --- title: http_client latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/http_client page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/http_client.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/http_client.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/http_client/)[Output](/redpanda-connect/components/outputs/http_client/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/http_client/ "View the Cloud version of this component"), Self-Managed Connects to a server and continuously requests single messages. #### Common ```yml inputs: label: "" http_client: url: "" # No default (required) verb: GET headers: {} rate_limit: "" # No default (optional) timeout: 5s payload: "" # No default (optional) stream: enabled: false reconnect: true scanner: lines: {} auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" http_client: url: "" # No default (required) verb: GET headers: {} metadata: include_prefixes: [] include_patterns: [] dump_request_log_level: "" oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" oauth2: enabled: false client_key: "" client_secret: "" token_url: "" scopes: [] endpoint_params: {} basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] extract_headers: include_prefixes: [] include_patterns: [] rate_limit: "" # No default (optional) timeout: 5s retry_period: 1s max_retry_backoff: 300s retries: 3 follow_redirects: true backoff_on: - 429 drop_on: [] successful_on: [] proxy_url: "" # No default (optional) disable_http2: false payload: "" # No default (optional) drop_empty_bodies: true stream: enabled: false reconnect: true scanner: lines: {} auto_replay_nacks: true ``` ## [](#dynamic-url-and-header-settings)Dynamic URL and header settings You can set the [`url`](#url) and [`headers`](#headers) values dynamically using [function interpolations](../../../configuration/interpolation/#bloblang-queries). You can also add [function interpolations](../../../configuration/interpolation/#bloblang-queries) to the [`url`](#url) and [`headers`](#headers) fields to implement basic pagination, such as page numbers or tokens, where subsequent requests need to include data from previously-consumed responses. Example: ```yaml input: http_client: url: >- https://api.example.com/search?query=allmyfoos&start_time=${! ( (timestamp_unix()-300).ts_format("2006-01-02T15:04:05Z","UTC").escape_url_query() ) }${! ("&next_token="+this.meta.next_token.not_null()) | "" } verb: GET rate_limit: schedule_searches oauth2: enabled: true token_url: https://api.example.com/oauth2/token client_key: "${EXAMPLE_KEY}" client_secret: "${EXAMPLE_SECRET}" rate_limit_resources: - label: schedule_searches local: count: 1 interval: 30s ``` > 💡 **TIP** > > If pagination requires more complex logic, consider using the [`http` processor](../../processors/http/) combined with a [`generate` input](../generate/), which allows you to schedule the processor. ## [](#streaming-messages)Streaming messages If you [enable streaming](#stream-enabled), Redpanda Connect consumes the body of the server response as a continuous stream of data, and breaks the stream down into smaller, logical messages using the [specified scanner](#stream-scanner). This functionality allows you to consume APIs that provide long-lived streamed data feeds, such as stock market feeds. ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#backoff_on)`backoff_on[]` A list of status codes that indicate a request failure, and trigger retries with an increasing backoff period between attempts. **Type**: `int` **Default**: ```yaml - 429 ``` ### [](#basic_auth)`basic_auth` Allows you to specify basic authentication. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#disable_http2)`disable_http2` Whether to disable HTTP/2. By default, HTTP/2 is enabled. Requires version 4.44.0 or later. **Type**: `bool` **Default**: `false` ### [](#drop_empty_bodies)`drop_empty_bodies` Whether to drop empty payloads received from the target server. **Type**: `bool` **Default**: `true` ### [](#drop_on)`drop_on[]` A list of status codes that indicate a request failure, where the input should not attempt retries. This helps avoid unnecessary retries for requests that are unlikely to succeed. > 📝 **NOTE** > > In these cases, the _request_ is dropped, but the _message_ that triggered the request is retained. **Type**: `int` **Default**: `[]` ### [](#dump_request_log_level)`dump_request_log_level` EXPERIMENTAL: Set the logging level for the request and response payloads of each HTTP request. Requires version 4.12.0 or later. **Type**: `string` **Default**: `""` **Options**: `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, `FATAL`, \`\` ### [](#extract_headers)`extract_headers` Specify which response headers to add to the resulting messages as metadata. Header keys are automatically converted to lowercase before matching, so make sure that your patterns target the lowercase versions of the expected header keys. **Type**: `object` ### [](#extract_headers-include_patterns)`extract_headers.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#extract_headers-include_prefixes)`extract_headers.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#follow_redirects)`follow_redirects` Whether or not to transparently follow redirects, i.e. responses with 300-399 status codes. If disabled, the response message will contain the body, status, and headers from the redirect response and the processor will not make a request to the URL set in the Location header of the response. **Type**: `bool` **Default**: `true` ### [](#headers)`headers` A map of headers to add to the request. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `{}` ```yaml # Examples: headers: Content-Type: application/octet-stream traceparent: ${! tracing_span().traceparent } ``` ### [](#jwt)`jwt` Beta Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWT tokens provide secure, stateless authentication between services. **Type**: `object` ### [](#jwt-claims)`jwt.claims` A value used to identify the claims that issued the JWT. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` The cryptographic algorithm used to sign the JWT token. Supported algorithms include RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#max_retry_backoff)`max_retry_backoff` The maximum period to wait between failed requests. **Type**: `string` **Default**: `300s` ### [](#metadata)`metadata` Specify matching rules that determine which metadata keys to add to the HTTP request as headers (optional). **Type**: `object` ### [](#metadata-include_patterns)`metadata.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#metadata-include_prefixes)`metadata.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#oauth)`oauth` Configure OAuth version 1.0 authentication for secure API access. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` The value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` The secret that establishes ownership of the `oauth.access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` A value used to identify the client to the service provider. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#oauth2)`oauth2` Allows you to specify open authentication using OAuth version 2 and the client credentials token flow. **Type**: `object` ### [](#oauth2-client_key)`oauth2.client_key` A value used to identify the client to the token provider. **Type**: `string` **Default**: `""` ### [](#oauth2-client_secret)`oauth2.client_secret` The secret used to establish ownership of the client key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth2-enabled)`oauth2.enabled` Whether to use OAuth version 2 in requests. **Type**: `bool` **Default**: `false` ### [](#oauth2-endpoint_params)`oauth2.endpoint_params` A list of endpoint parameters specified as arrays of strings (optional). Requires version 4.21.0 or later. **Type**: `object` **Default**: `{}` ```yaml # Examples: endpoint_params: bar: - woof foo: - meow - quack ``` ### [](#oauth2-scopes)`oauth2.scopes[]` A list of requested permissions (optional). Requires version 3.45.0 or later. **Type**: `array` **Default**: `[]` ### [](#oauth2-token_url)`oauth2.token_url` The URL of the token provider. **Type**: `string` **Default**: `""` ### [](#payload)`payload` A payload to deliver for each request (optional). This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#proxy_url)`proxy_url` A HTTP proxy URL (optional). **Type**: `string` ### [](#rate_limit)`rate_limit` A [rate limit](../../rate_limits/about/) to throttle requests by (optional). **Type**: `string` ### [](#retries)`retries` The maximum number of retry attempts to make. **Type**: `int` **Default**: `3` ### [](#retry_period)`retry_period` The initial period to wait between failed requests before retrying. **Type**: `string` **Default**: `1s` ### [](#stream)`stream` Enables streaming mode, where the HTTP connection remains open and messages are processed line-by-line. **Type**: `object` ### [](#stream-enabled)`stream.enabled` Enables streaming mode. **Type**: `bool` **Default**: `false` ### [](#stream-reconnect)`stream.reconnect` Whether to automatically reestablish the HTTP connection if it is lost. **Type**: `bool` **Default**: `true` ### [](#stream-scanner)`stream.scanner` The [scanner](../../scanners/about/) used to split the stream of bytes into individual messages. Scanners are useful for processing large data sources efficiently without holding the entire data set in memory. For example, the `csv` scanner processes individual rows in a CSV file without loading the entire file in memory. Requires version 4.25.0 or later. **Type**: `scanner` **Default**: ```yaml lines: {} ``` ### [](#successful_on)`successful_on[]` A list of HTTP status codes that should be considered as successful, even if they are not 2XX codes. This is useful for handling cases where non-2XX codes indicate that the request was processed successfully, such as `303 See Other` or `409 Conflict`. By default, all 2XX codes are considered successful unless they are specified in `backoff_on` or `drop_on` fields. **Type**: `int` **Default**: `[]` ### [](#timeout)`timeout` A static timeout to apply to requests. **Type**: `string` **Default**: `5s` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL to connect to. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#verb)`verb` A verb to connect with. **Type**: `string` **Default**: `GET` ```yaml # Examples: verb: POST # --- verb: GET # --- verb: DELETE ``` --- # Page 63: http_server **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/http_server.md --- # http\_server --- title: http_server latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/http_server page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/http_server.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/http_server.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-20" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/http_server/)[Output](/redpanda-connect/components/outputs/http_server/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/http_server/ "View the Cloud version of this component"), Self-Managed Receive messages sent over HTTP using POST requests. HTTP 2.0 is supported when using TLS, which is enabled when key and cert files are specified. #### Common ```yml inputs: label: "" http_server: address: "" path: /post ws_path: /post/ws allowed_verbs: - "POST" timeout: 5s rate_limit: "" ``` #### Advanced ```yml inputs: label: "" http_server: address: "" path: /post ws_path: /post/ws ws_welcome_message: "" ws_rate_limit_message: "" allowed_verbs: - "POST" timeout: 5s rate_limit: "" cert_file: "" key_file: "" cors: enabled: false allowed_origins: [] sync_response: status: 200 headers: Content-Type: "application/octet-stream" metadata_headers: include_prefixes: [] include_patterns: [] tcp: reuse_addr: false reuse_port: false ``` If the `address` config field is left blank, the [service-wide HTTP server](../../http/about/) is used. The field `rate_limit` allows you to specify an optional [`rate_limit` resource](../../rate_limits/about/), which will be applied to each HTTP request made and each websocket payload received. When the rate limit is breached HTTP requests will have a 429 response returned with a Retry-After header. Websocket payloads will be dropped and an optional response payload will be sent as per `ws_rate_limit_message`. ## [](#responses)Responses It’s possible to return a response for each message received using [synchronous responses](../../../guides/sync_responses/). When doing so you can customize headers with the `sync_response` field `headers`, which can also use [function interpolation](../../../configuration/interpolation/#bloblang-queries) in the value based on the response message contents. ## [](#endpoints)Endpoints The following fields specify endpoints that are registered for sending messages, and support path parameters of the form `/{foo}`, which are added to ingested messages as metadata. A path ending in `/` will match against all extensions of that path: ### [](#path-defaults-to-post)`path` (defaults to `/post`) This endpoint expects POST requests where the entire request body is consumed as a single message. If the request contains a multipart `content-type` header as per [RFC1341](https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html) then the multiple parts are consumed as a batch of messages, where each body part is a message of the batch. ### [](#ws_path-defaults-to-postws)`ws_path` (defaults to `/post/ws`) Creates a websocket connection, where payloads received on the socket are passed through the pipeline as a batch of one message. > ⚠️ **CAUTION: Endpoint caveats** > > Endpoint caveats > > Components within a Redpanda Connect config will register their respective endpoints in a non-deterministic order. This means that establishing precedence of endpoints that are registered via multiple `http_server` inputs or outputs (either within brokers or from cohabiting streams) is not possible in a predictable way. > > This ambiguity makes it difficult to ensure that paths which are both a subset of a path registered by a separate component, and end in a slash (`/`) and will therefore match against all extensions of that path, do not prevent the more specific path from matching against requests. > > It is therefore recommended that you ensure paths of separate components do not collide unless they are explicitly non-competing. > > For example, if you were to deploy two separate `http_server` inputs, one with a path `/foo/` and the other with a path `/foo/bar`, it would not be possible to ensure that the path `/foo/` does not swallow requests made to `/foo/bar`. You may specify an optional `ws_welcome_message`, which is a static payload to be sent to all clients once a websocket connection is first established. It’s also possible to specify a `ws_rate_limit_message`, which is a static payload to be sent to clients that have triggered the servers rate limit. ## [](#metadata)Metadata This input adds the following metadata fields to each message: ```text - http_server_user_agent - http_server_request_path - http_server_verb - http_server_remote_ip - All headers (only first values are taken) - All query parameters - All path parameters - All cookies ``` If HTTPS is enabled, the following fields are added as well: ```text - http_server_tls_version - http_server_tls_subject - http_server_tls_cipher_suite ``` You can access these metadata fields using [function interpolation](../../../configuration/interpolation/#bloblang-queries). ### [](#headers)Headers Request headers are available as metadata and use the HTTP header name with no additional prefix as a key. During processing, Redpanda Connect changes the format of the header name, as in the following example: ```text x-api-key available as metadata("X-Api-Key") ``` ## [](#examples)Examples ### [](#path-switching)Path Switching This example shows an `http_server` input that captures all requests and processes them by switching on that path: ```yaml input: http_server: path: / allowed_verbs: [ GET, POST ] sync_response: headers: Content-Type: application/json processors: - switch: - check: '@http_server_request_path == "/foo"' processors: - mapping: | root.title = "You Got Fooed!" root.result = content().string().uppercase() - check: '@http_server_request_path == "/bar"' processors: - mapping: 'root.title = "Bar Is Slow"' - sleep: # Simulate a slow endpoint duration: 1s ``` ### [](#mock-oauth-2-0-server)Mock OAuth 2.0 Server This example shows an `http_server` input that mocks an OAuth 2.0 Client Credentials flow server at the endpoint `/oauth2_test`: ```yaml input: http_server: path: /oauth2_test allowed_verbs: [ GET, POST ] sync_response: headers: Content-Type: application/json processors: - log: message: "Received request" level: INFO fields_mapping: | root = @ root.body = content().string() - mapping: | root.access_token = "MTQ0NjJkZmQ5OTM2NDE1ZTZjNGZmZjI3" root.token_type = "Bearer" root.expires_in = 3600 - sync_response: {} - mapping: 'root = deleted()' ``` ## [](#fields)Fields ### [](#address)`address` An alternative address to host from. If left empty the service wide address is used. **Type**: `string` **Default**: `""` ### [](#allowed_verbs)`allowed_verbs[]` An array of verbs that are allowed for the `path` endpoint. Requires version 3.33.0 or later. **Type**: `array` **Default**: ```yaml - "POST" ``` ### [](#cert_file)`cert_file` Enable TLS by specifying a certificate and key file. Only valid with a custom `address`. **Type**: `string` **Default**: `""` ### [](#cors)`cors` Adds Cross-Origin Resource Sharing headers. Only valid with a custom `address`. Requires version 3.63.0 or later. **Type**: `object` ### [](#cors-allowed_origins)`cors.allowed_origins[]` An explicit list of origins that are allowed for CORS requests. **Type**: `array` **Default**: `[]` ### [](#cors-enabled)`cors.enabled` Whether to allow CORS requests. **Type**: `bool` **Default**: `false` ### [](#key_file)`key_file` Enable TLS by specifying a certificate and key file. Only valid with a custom `address`. **Type**: `string` **Default**: `""` ### [](#path)`path` The endpoint path to listen for POST requests. **Type**: `string` **Default**: `/post` ### [](#rate_limit)`rate_limit` An optional [rate limit](../../rate_limits/about/) to throttle requests by. **Type**: `string` **Default**: `""` ### [](#sync_response)`sync_response` Customize messages returned via [synchronous responses](../../../guides/sync_responses/). **Type**: `object` ### [](#sync_response-headers)`sync_response.headers` Specify headers to return with synchronous responses. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: ```yaml Content-Type: "application/octet-stream" ``` ### [](#sync_response-metadata_headers)`sync_response.metadata_headers` Specify criteria for which metadata values are added to the response as headers. **Type**: `object` ### [](#sync_response-metadata_headers-include_patterns)`sync_response.metadata_headers.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#sync_response-metadata_headers-include_prefixes)`sync_response.metadata_headers.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#sync_response-status)`sync_response.status` Specify the status code to return with synchronous responses. This is a string value, which allows you to customize it based on resulting payloads and their metadata. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `200` ```yaml # Examples: status: ${! json("status") } # --- status: ${! meta("status") } ``` ### [](#tcp)`tcp` TCP listener configuration for the HTTP server. Only valid with a custom `address`. **Type**: `object` ### [](#tcp-reuse_addr)`tcp.reuse_addr` Enable SO\_REUSEADDR, allowing binding to ports in TIME\_WAIT state. Useful for graceful restarts and config reloads where the server needs to rebind to the same port immediately after shutdown. **Type**: `bool` **Default**: `false` ### [](#tcp-reuse_port)`tcp.reuse_port` Enable SO\_REUSEPORT, allowing multiple sockets to bind to the same port for load balancing across multiple processes/threads. **Type**: `bool` **Default**: `false` ### [](#timeout)`timeout` Timeout for requests. If a consumed messages takes longer than this to be delivered the connection is closed, but the message may still be delivered. **Type**: `string` **Default**: `5s` ### [](#ws_path)`ws_path` The endpoint path to create websocket connections from. **Type**: `string` **Default**: `/post/ws` ### [](#ws_rate_limit_message)`ws_rate_limit_message` An optional message to delivery to websocket connections that are rate limited. **Type**: `string` **Default**: `""` ### [](#ws_welcome_message)`ws_welcome_message` An optional message to deliver to fresh websocket connections. **Type**: `string` **Default**: `""` --- # Page 64: inproc **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/inproc.md --- # inproc --- title: inproc latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/inproc page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/inproc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/inproc.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/inproc/)[Output](/redpanda-connect/components/outputs/inproc/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/inproc/ "View the Cloud version of this component"), Self-Managed ```yml inputs: label: "" inproc: "" ``` Directly connect to an output within a Redpanda Connect process by referencing it by a chosen ID. This allows you to hook up isolated streams whilst running Redpanda Connect in [streams mode](../../../guides/streams_mode/about/), it is NOT recommended that you connect the inputs of a stream with an output of the same stream, as feedback loops can lead to deadlocks in your message flow. It is possible to connect multiple inputs to the same inproc ID, resulting in messages dispatching in a round-robin fashion to connected inputs. However, only one output can assume an inproc ID, and will replace existing outputs if a collision occurs. --- # Page 65: kafka_franz **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/kafka_franz.md --- # kafka\_franz --- title: kafka_franz latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/kafka_franz page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/kafka_franz.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/kafka_franz.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/kafka_franz/)[Output](/redpanda-connect/components/outputs/kafka_franz/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/kafka_franz/ "View the Cloud version of this component"), Self-Managed > ⚠️ **WARNING: Deprecated in 4.68.0** > > Deprecated in 4.68.0 > > This component is deprecated and will be removed in the next major version release. Please consider moving onto the unified [`redpanda` input](../redpanda/) and [`redpanda` output](../../outputs/redpanda/) components. A Kafka input using the [Franz Kafka client library](https://github.com/twmb/franz-go). Introduced in version 3.61.0. #### Common ```yml inputs: label: "" kafka_franz: seed_brokers: [] # No default (required) topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" kafka_franz: seed_brokers: [] # No default (required) client_id: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) metadata_max_age: 1m request_timeout_overhead: 10s conn_idle_timeout: 20s tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) rack_id: "" instance_id: "" rebalance_timeout: 45s session_timeout: 1m heartbeat_interval: 3s start_offset: earliest fetch_max_bytes: 50MiB fetch_max_wait: 5s fetch_min_bytes: 1B fetch_max_partition_bytes: 1MiB transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) checkpoint_limit: 1024 commit_period: 5s multi_header: false batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) topic_lag_refresh_period: 5s auto_replay_nacks: true timely_nacks_maximum_wait: "" # No default (optional) ``` When you specify a consumer group in your configuration, this input consumes one or more topics and automatically balances the topic partitions across any other connected clients with the same consumer group. Otherwise, topics are consumed in their entirety or with explicit partitions. This input often out-performs the traditional `kafka` input and provides more useful logs and error messages. ## [](#metadata)Metadata This input adds the following metadata fields to each message: ```text - kafka_key - kafka_topic - kafka_partition - kafka_offset - kafka_timestamp_ms - kafka_timestamp_unix - kafka_tombstone_message - All record headers ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements) at the output level. If the cause of rejections persists, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#batching)`batching` Configure a [batching policy](../../../configuration/batching/) that applies to individual topic partitions in order to batch messages together before flushing them for processing. Batching can be beneficial for performance as well as useful for windowed processing, and doing so this way preserves the ordering of topic partitions. **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#checkpoint_limit)`checkpoint_limit` The maximum number of messages that are processed in parallel inside the same partition before back pressure is applied. When a message with a specific offset is delivered to the output, the offset is only committed when all messages of previous offsets have also been delivered. This behavior ensures at-least-once delivery guarantees. However, in the event of crashes or server faults, it also increases the likelihood of duplicates. To decrease this risk, reduce the `checkpoint_limit` value. **Type**: `int` **Default**: `1024` ### [](#client_id)`client_id` An identifier for the client connection. **Type**: `string` **Default**: `redpanda-connect` ### [](#commit_period)`commit_period` The period of time between each commit of the current partition offsets. Offsets are always committed during shutdown. **Type**: `string` **Default**: `5s` ### [](#conn_idle_timeout)`conn_idle_timeout` The maximum duration that connections can remain idle before they are automatically closed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `20s` ### [](#consumer_group)`consumer_group` An optional consumer group. When you specify this value: - The partitions of any topics, specified in the `topics` field, are automatically distributed across consumers sharing a consumer group - Partition offsets are automatically committed and resumed under this name Consumer groups are not supported when you specify explicit partitions to consume from in the `topics` field. **Type**: `string` ### [](#fetch_max_bytes)`fetch_max_bytes` The maximum size of a message batch (in bytes) that a broker tries to send during a client fetch. If individual records exceed the `fetch_max_bytes` value, brokers will still send them. **Type**: `string` **Default**: `50MiB` ### [](#fetch_max_partition_bytes)`fetch_max_partition_bytes` The maximum number of bytes that are consumed from a single partition in a fetch request. This field is equivalent to the Java setting `fetch.max.partition.bytes`. If a single batch is larger than the `fetch_max_partition_bytes` value, the batch is still sent so that the client can make progress. **Type**: `string` **Default**: `1MiB` ### [](#fetch_max_wait)`fetch_max_wait` The maximum period of time a broker can wait for a fetch response to reach the required minimum number of bytes (`fetch_min_bytes`). **Type**: `string` **Default**: `5s` ### [](#fetch_min_bytes)`fetch_min_bytes` The minimum number of bytes that a broker tries to send during a fetch. This field is equivalent to the Java setting `fetch.min.bytes`. **Type**: `string` **Default**: `1B` ### [](#heartbeat_interval)`heartbeat_interval` When you specify a `consumer_group`, `heartbeat_interval` sets how frequently a consumer group member should send heartbeats to Apache Kafka. Apache Kafka uses heartbeats to make sure that a group member’s session is active. You must set `heartbeat_interval` to less than one-third of `session_timeout`. This field is equivalent to the Java `heartbeat.interval.ms` setting and accepts Go duration format strings such as `10s` or `2m`. **Type**: `string` **Default**: `3s` ### [](#instance_id)`instance_id` When you specify a [`consumer_group`](#consumer_group), assign a unique value to `instance_id` to define the group’s static membership, which can prevent unnecessary rebalances during reconnections. When you assign an instance ID, the client does not automatically leave the consumer group when it disconnects. To remove the client, you must use an external admin command on behalf of the instance ID. **Type**: `string` **Default**: `""` ### [](#metadata_max_age)`metadata_max_age` The maximum period of time after which metadata is refreshed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Lower values provide more responsive topic and partition discovery but may increase broker load. Higher values reduce broker queries but can delay detection of topology changes. **Type**: `string` **Default**: `1m` ### [](#multi_header)`multi_header` Decode headers into lists to allow the handling of multiple values with the same key. **Type**: `bool` **Default**: `false` ### [](#rack_id)`rack_id` A rack specifies where the client is physically located, and changes fetch requests to consume from the closest replica as opposed to the leader replica. **Type**: `string` **Default**: `""` ### [](#rebalance_timeout)`rebalance_timeout` When you specify a [`consumer_group`](#consumer_group), `rebalance_timeout` sets a time limit for all consumer group members to complete their work and commit offsets after a rebalance has begun. The timeout excludes the time taken to detect a failed or late heartbeat, which indicates a rebalance is required. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `45s` ### [](#regexp_topics_exclude)`regexp_topics_exclude[]` A list of regular expression patterns for excluding topics when regex mode is enabled (using `regexp_topics_include` or the deprecated `regexp_topics` boolean). Topics matching any of these patterns will be excluded from consumption, even if they match include patterns. Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so use `^` and `$` for exact matching. Exclude patterns are applied after include patterns, providing fine-grained control over topic selection. Example: `regexp_topics_exclude: ["^_", ".**-temp$", ".**-test.*"]` excludes topics starting with underscore, ending with `-temp`, or containing `-test`. **Type**: `array` ### [](#regexp_topics_include)`regexp_topics_include[]` A list of regular expression patterns for matching topics to consume from. When specified, the client will periodically refresh the list of matching topics based on the `metadata_max_age` interval. Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so `logs_.` **matches `my-logs_events` and `logs_errors`. Use `^logs_.`**`$` to match only topics starting with `logs_`. This field enables regex mode (replacing the deprecated `regexp_topics` boolean) and cannot be used together with explicit `topics` lists. Use `regexp_topics_exclude` to filter out specific patterns from the matched topics. Example: `regexp_topics_include: ["events_.**", "logs_.**"]` consumes from all topics starting with `events_` or `logs_`. **Type**: `array` ```yaml # Examples: regexp_topics_include: - logs_.* - metrics_.* # --- regexp_topics_include: - "events_[0-9]+" ``` ### [](#request_timeout_overhead)`request_timeout_overhead` Grants an additional buffer or overhead to requests that have timeout fields defined. This field is based on the behavior of Apache Kafka’s `request.timeout.ms` parameter. **Type**: `string` **Default**: `10s` ### [](#sasl)`sasl[]` Specify one or more methods or mechanisms of SASL authentication, which are attempted in order. If the broker supports the first SASL mechanism, all connections use it. If the first mechanism fails, the client picks the first supported mechanism. If the broker does not support any client mechanisms, all connections fail. **Type**: `object` ```yaml # Examples: sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ### [](#sasl-aws)`sasl[].aws` Contains AWS specific fields for when the `mechanism` is set to `AWS_MSK_IAM`. **Type**: `object` ### [](#sasl-aws-credentials)`sasl[].aws.credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](../../../guides/cloud/aws/). **Type**: `object` ### [](#sasl-aws-credentials-from_ec2_role)`sasl[].aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#sasl-aws-credentials-id)`sasl[].aws.credentials.id` The ID of credentials to use. **Type**: `string` ### [](#sasl-aws-credentials-profile)`sasl[].aws.credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#sasl-aws-credentials-role)`sasl[].aws.credentials.role` A role ARN to assume. **Type**: `string` ### [](#sasl-aws-credentials-role_external_id)`sasl[].aws.credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#sasl-aws-credentials-secret)`sasl[].aws.credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#sasl-aws-credentials-token)`sasl[].aws.credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#sasl-aws-endpoint)`sasl[].aws.endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#sasl-aws-region)`sasl[].aws.region` The AWS region to target. **Type**: `string` ### [](#sasl-aws-tcp)`sasl[].aws.tcp` TCP socket configuration. **Type**: `object` ### [](#sasl-aws-tcp-connect_timeout)`sasl[].aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-aws-tcp-keep_alive)`sasl[].aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#sasl-aws-tcp-keep_alive-count)`sasl[].aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#sasl-aws-tcp-keep_alive-idle)`sasl[].aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-keep_alive-interval)`sasl[].aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-tcp_user_timeout)`sasl[].aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-extensions)`sasl[].extensions` Key/value pairs to add to OAUTHBEARER authentication requests. **Type**: `string` ### [](#sasl-mechanism)`sasl[].mechanism` The SASL mechanism to use. **Type**: `string` | Option | Summary | | --- | --- | | AWS_MSK_IAM | AWS IAM based authentication as specified by the 'aws-msk-iam-auth' java library. | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. | | REDPANDA_CLOUD_SERVICE_ACCOUNT | Redpanda Cloud Service Account authentication when running in Redpanda Cloud. | | SCRAM-SHA-256 | SCRAM based authentication as specified in RFC5802. | | SCRAM-SHA-512 | SCRAM based authentication as specified in RFC5802. | | none | Disable sasl authentication | ### [](#sasl-password)`sasl[].password` A password to provide for PLAIN or SCRAM-\* authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#sasl-token)`sasl[].token` The token to use for a single session’s OAUTHBEARER authentication. **Type**: `string` **Default**: `""` ### [](#sasl-username)`sasl[].username` A username to provide for PLAIN or SCRAM-\* authentication. **Type**: `string` **Default**: `""` ### [](#seed_brokers)`seed_brokers[]` A list of broker addresses to connect to in order. Use commas to separate multiple addresses in a single list item. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" # --- seed_brokers: - "foo:9092" - "bar:9092" # --- seed_brokers: - "foo:9092,bar:9092" ``` ### [](#session_timeout)`session_timeout` When you specify a `consumer_group`, `session_timeout` sets the maximum interval between heartbeats sent by a consumer group member to the broker. If a broker doesn’t receive a heartbeat from a group member before the timeout expires, it removes the member from the consumer group and initiates a rebalance. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `1m` ### [](#start_offset)`start_offset` Specify the offset from which this input starts or restarts consuming messages. Restarts occur when the `OffsetOutOfRange` error is seen during a fetch. **Type**: `string` **Default**: `earliest` | Option | Summary | | --- | --- | | committed | Prevents consuming a partition in a group if the partition has no prior commits. Corresponds to Kafka’s auto.offset.reset=none option | | earliest | Start from the earliest offset. Corresponds to Kafka’s auto.offset.reset=earliest option. | | latest | Start from the latest offset. Corresponds to Kafka’s auto.offset.reset=latest option. | ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timely_nacks_maximum_wait)`timely_nacks_maximum_wait` EXPERIMENTAL: Specify a maximum period of time in which each message can be consumed and awaiting either acknowledgement or rejection before rejection is instead forced. This can be useful for avoiding situations where certain downstream components can result in blocked confirmation of delivery that exceeds SLAs. Accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic_lag_refresh_period)`topic_lag_refresh_period` The interval between refresh cycles. During each cycle, this input queries the Redpanda Connect server to calculate the topic lag minus the number of produced messages that remain to be read from each topic/partition pair by the specified consumer group. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `5s` ### [](#topics)`topics[]` A list of topics to consume from. Use commas to separate multiple topics in a single element. When a `consumer_group` is specified, partitions are automatically distributed across consumers of a topic. Otherwise, all partitions are consumed. Alternatively, you can specify explicit partitions to consume by using a colon after the topic name. For example, `foo:0` would consume the partition `0` of the topic foo. This syntax supports ranges. For example, `foo:0-10` would consume partitions `0` through to `10` inclusive. It is also possible to specify an explicit offset to consume from by adding another colon after the partition. For example, `foo:0:10` would consume the partition `0` of the topic `foo` starting from the offset `10`. If the offset is not present (or remains unspecified) then the field `start_offset` determines which offset to start from. **Type**: `array` ```yaml # Examples: topics: - foo - bar # --- topics: - things.* # --- topics: - "foo,bar" # --- topics: - "foo:0" - "bar:1" - "bar:3" # --- topics: - "foo:0,bar:1,bar:3" # --- topics: - "foo:0-5" ``` ### [](#transaction_isolation_level)`transaction_isolation_level` The isolation level for handling transactional messages. This setting determines how transactions are processed and affects data consistency guarantees. **Type**: `string` **Default**: `read_uncommitted` | Option | Summary | | --- | --- | | read_committed | If set, only committed transactional records are processed. | | read_uncommitted | If set, then uncommitted records are processed. | --- # Page 66: kafka **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/kafka.md --- # kafka --- title: kafka latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/kafka page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/kafka.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/kafka.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/kafka/)[Output](/redpanda-connect/components/outputs/kafka/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/kafka/ "View the Cloud version of this component"), Self-Managed > ⚠️ **WARNING: Deprecated in 4.68.0** > > Deprecated in 4.68.0 > > This component is deprecated and will be removed in the next major version release. Please consider moving onto the unified [`redpanda` input](../redpanda/) and [`redpanda` output](../../outputs/redpanda/) components. Connects to Kafka brokers and consumes one or more topics. #### Common ```yml inputs: label: "" kafka: addresses: [] # No default (required) topics: [] # No default (required) target_version: "" # No default (optional) consumer_group: "" checkpoint_limit: 1024 auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" kafka: addresses: [] # No default (required) topics: [] # No default (required) target_version: "" # No default (optional) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: mechanism: none user: "" password: "" access_token: "" token_cache: "" token_key: "" consumer_group: "" client_id: benthos instance_id: "" # No default (optional) rack_id: "" start_from_oldest: true checkpoint_limit: 1024 auto_replay_nacks: true timely_nacks_maximum_wait: "" # No default (optional) commit_period: 1s max_processing_period: 100ms extract_tracing_map: "" # No default (optional) group: session_timeout: 10s heartbeat_interval: 3s rebalance_timeout: 60s fetch_buffer_cap: 256 multi_header: false batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` Offsets are managed within Kafka under the specified consumer group, and partitions for each topic are automatically balanced across members of the consumer group. The Kafka input allows parallel processing of messages from different topic partitions, and messages of the same topic partition are processed with a maximum parallelism determined by the field [`checkpoint_limit`](#checkpoint_limit). To enforce ordered processing of partition messages, set the [`checkpoint_limit`](#checkpoint_limit) to `1`, which makes sure that a message is only processed after the previous message is delivered. Batching messages before processing can be enabled using the [`batching`](#batching) field, and this batching is performed per-partition such that messages of a batch will always originate from the same partition. This batching mechanism is capable of creating batches of greater size than the [`checkpoint_limit`](#checkpoint_limit), in which case the next batch will only be created upon delivery of the current one. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - kafka\_key - kafka\_topic - kafka\_partition - kafka\_offset - kafka\_lag - kafka\_timestamp\_ms - kafka\_timestamp\_unix - kafka\_tombstone\_message - All existing message headers (version 0.11+) The field `kafka_lag` is the calculated difference between the high water mark offset of the partition at the time of ingestion and the current message offset. You can access these metadata fields using [function interpolation](../../../configuration/interpolation/#bloblang-queries). ## [](#ordering)Ordering By default messages of a topic partition can be processed in parallel, up to a limit determined by the field `checkpoint_limit`. However, if strict ordered processing is required then this value must be set to 1 in order to process shard messages in lock-step. When doing so it is recommended that you perform batching at this component for performance as it will not be possible to batch lock-stepped messages at the output level. ## [](#troubleshooting)Troubleshooting If you’re seeing issues writing to or reading from Kafka with this component then it’s worth trying out the newer [`kafka_franz` input](../kafka_franz/). - I’m seeing logs that report `Failed to connect to kafka: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)`, but the brokers are definitely reachable. Unfortunately this error message will appear for a wide range of connection problems even when the broker endpoint can be reached. Double check your authentication configuration and also ensure that you have [enabled TLS](#tlsenabled) if applicable. ## [](#fields)Fields ### [](#addresses)`addresses[]` A list of broker addresses to connect to. If an item of the list contains commas it will be expanded into multiple addresses. **Type**: `array` ```yaml # Examples: addresses: - "localhost:9092" # --- addresses: - "localhost:9041,localhost:9042" # --- addresses: - "localhost:9041" - "localhost:9042" ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#checkpoint_limit)`checkpoint_limit` The maximum number of messages of the same topic and partition that can be processed at a given time. Increasing this limit enables parallel processing and batching at the output level to work on individual partitions. Any given offset will not be committed unless all messages under that offset are delivered in order to preserve at least once delivery guarantees. Requires version 3.33.0 or later. **Type**: `int` **Default**: `1024` ### [](#client_id)`client_id` An identifier for the client connection. **Type**: `string` **Default**: `benthos` ### [](#commit_period)`commit_period` The period of time between each commit of the current partition offsets. Offsets are always committed during shutdown. **Type**: `string` **Default**: `1s` ### [](#consumer_group)`consumer_group` An identifier for the consumer group of the connection. This field can be explicitly made empty in order to disable stored offsets for the consumed topic partitions. **Type**: `string` **Default**: `""` ### [](#extract_tracing_map)`extract_tracing_map` EXPERIMENTAL: A [Bloblang mapping](../../../guides/bloblang/about/) that attempts to extract an object containing tracing propagation information, which will then be used as the root tracing span for the message. The specification of the extracted fields must match the format used by the service wide tracer. Requires version 3.45.0 or later. **Type**: `string` ```yaml # Examples: extract_tracing_map: root = @ # --- extract_tracing_map: root = this.meta.span ``` ### [](#fetch_buffer_cap)`fetch_buffer_cap` The maximum number of unprocessed messages to fetch at a given time. **Type**: `int` **Default**: `256` ### [](#group)`group` Tuning parameters for consumer group synchronization. **Type**: `object` ### [](#group-heartbeat_interval)`group.heartbeat_interval` A period in which heartbeats should be sent out. **Type**: `string` **Default**: `3s` ### [](#group-rebalance_timeout)`group.rebalance_timeout` A period after which rebalancing is abandoned if unresolved. **Type**: `string` **Default**: `60s` ### [](#group-session_timeout)`group.session_timeout` A period after which a consumer of the group is kicked after no heartbeats. **Type**: `string` **Default**: `10s` ### [](#instance_id)`instance_id` When you specify a [`consumer_group`](#consumer_group), assign a unique value to `instance_id` to help brokers identify each input after restarts and prevent unnecessary rebalances. **Type**: `string` ### [](#max_processing_period)`max_processing_period` A maximum estimate for the time taken to process a message, this is used for tuning consumer group synchronization. **Type**: `string` **Default**: `100ms` ### [](#multi_header)`multi_header` Decode headers into lists to allow handling of multiple values with the same key **Type**: `bool` **Default**: `false` ### [](#rack_id)`rack_id` A rack identifier for this client. **Type**: `string` **Default**: `""` ### [](#sasl)`sasl` Enables SASL authentication. **Type**: `object` ### [](#sasl-access_token)`sasl.access_token` A static OAUTHBEARER access token **Type**: `string` **Default**: `""` ### [](#sasl-mechanism)`sasl.mechanism` The SASL authentication mechanism, if left empty SASL authentication is not used. **Type**: `string` **Default**: `none` | Option | Summary | | --- | --- | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. NOTE: When using plain text auth it is extremely likely that you’ll also need to enable TLS. | | SCRAM-SHA-256 | Authentication using the SCRAM-SHA-256 mechanism. | | SCRAM-SHA-512 | Authentication using the SCRAM-SHA-512 mechanism. | | none | Default, no SASL authentication. | ### [](#sasl-password)`sasl.password` A PLAIN password. It is recommended that you use environment variables to populate this field. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: ${PASSWORD} ``` ### [](#sasl-token_cache)`sasl.token_cache` Instead of using a static `access_token` allows you to query a [`cache`](../../caches/about/) resource to fetch OAUTHBEARER tokens from **Type**: `string` **Default**: `""` ### [](#sasl-token_key)`sasl.token_key` Required when using a `token_cache`, the key to query the cache with for tokens. **Type**: `string` **Default**: `""` ### [](#sasl-user)`sasl.user` A PLAIN username. It is recommended that you use environment variables to populate this field. **Type**: `string` **Default**: `""` ```yaml # Examples: user: ${USER} ``` ### [](#start_from_oldest)`start_from_oldest` Determines whether to consume from the oldest available offset, otherwise messages are consumed from the latest offset. The setting is applied when creating a new consumer group or the saved offset no longer exists. **Type**: `bool` **Default**: `true` ### [](#target_version)`target_version` The version of the Kafka protocol to use. This limits the capabilities used by the client and should ideally match the version of your brokers. Defaults to the oldest supported stable version. **Type**: `string` ```yaml # Examples: target_version: 2.1.0 # --- target_version: 3.1.0 ``` ### [](#timely_nacks_maximum_wait)`timely_nacks_maximum_wait` EXPERIMENTAL: Specify a maximum period of time in which each message can be consumed and awaiting either acknowledgement or rejection before rejection is instead forced. This can be useful for avoiding situations where certain downstream components can result in blocked confirmation of delivery that exceeds SLAs. Accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#topics)`topics[]` A list of topics to consume from. Multiple comma separated topics can be listed in a single element. Partitions are automatically distributed across consumers of a topic. Alternatively, it’s possible to specify explicit partitions to consume from with a colon after the topic name, e.g. `foo:0` would consume the partition 0 of the topic foo. This syntax supports ranges, e.g. `foo:0-10` would consume partitions 0 through to 10 inclusive. Requires version 3.33.0 or later. **Type**: `array` ```yaml # Examples: topics: - foo - bar # --- topics: - "foo,bar" # --- topics: - "foo:0" - "bar:1" - "bar:3" # --- topics: - "foo:0,bar:1,bar:3" # --- topics: - "foo:0-5" ``` --- # Page 67: microsoft_sql_server_cdc **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/microsoft_sql_server_cdc.md --- # microsoft\_sql\_server\_cdc --- title: microsoft_sql_server_cdc latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/microsoft_sql_server_cdc page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/microsoft_sql_server_cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/microsoft_sql_server_cdc.adoc description: Enables Change Data Capture by consuming from Microsoft SQL Server's change tables. page-git-created-date: "2025-10-24" page-git-modified-date: "2025-10-24" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/microsoft_sql_server_cdc/ "View the Cloud version of this component"), Self-Managed **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Enables Change Data Capture by consuming from Microsoft SQL Server’s change tables. Introduced in version 4.67.5. #### Common ```yaml inputs: label: "" microsoft_sql_server_cdc: connection_string: "" # No default (required) stream_snapshot: false max_parallel_snapshot_tables: 1 snapshot_max_batch_size: 1000 include: [] # No default (required) exclude: [] # No default (optional) checkpoint_cache: "" # No default (optional) checkpoint_cache_table_name: rpcn.CdcCheckpointCache checkpoint_cache_connection_string: "" # No default (optional) checkpoint_cache_key: microsoft_sql_server_cdc checkpoint_limit: 1024 stream_backoff_interval: 5s auto_replay_nacks: true batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yaml inputs: label: "" microsoft_sql_server_cdc: connection_string: "" # No default (required) stream_snapshot: false max_parallel_snapshot_tables: 1 snapshot_max_batch_size: 1000 include: [] # No default (required) exclude: [] # No default (optional) checkpoint_cache: "" # No default (optional) checkpoint_cache_table_name: rpcn.CdcCheckpointCache checkpoint_cache_connection_string: "" # No default (optional) checkpoint_cache_key: microsoft_sql_server_cdc checkpoint_limit: 1024 stream_backoff_interval: 5s auto_replay_nacks: true batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` Streams changes from a Microsoft SQL Server database for Change Data Capture (CDC). Additionally, if `stream_snapshot` is set to true, then the existing data in the database is also streamed too. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - schema (Schema of the table that the message originated from) - table (Name of the table that the message originated from) - operation (Type of operation that generated the message: "read", "delete", "insert", or "update\_before" and "update\_after". "read" is from messages that are read in the initial snapshot phase.) - lsn (the Log Sequence Number in Microsoft SQL Server) ## [](#permissions)Permissions To use the default Microsoft SQL Server cache, the user must have permissions to create tables and stored procedures. Refer to [`checkpoint_cache_table_name`](#checkpoint_cache_table_name) for additional details. ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay messages that are rejected (nacked) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#batching)`batching` Configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#checkpoint_cache)`checkpoint_cache` A [cache resource](../../caches/about/) to store the current Log Sequence Number (LSN) position. This enables the connector to resume from the last processed position after restarts, preventing data loss and duplicate processing. The cache stores the highest LSN that has been successfully delivered downstream. **Type**: `string` ### [](#checkpoint_cache_connection_string)`checkpoint_cache_connection_string` An optional connection string for a remote Microsoft SQL Server to use for the checkpoint cache. When set, this creates the checkpoint cache table on the remote server instead of the source database. If `checkpoint_cache` is also set, that takes precedence. **Type**: `string` ```yaml # Examples: checkpoint_cache_connection_string: sqlserver://username:password@remotehost/instance?param1=value¶m2=value ``` ### [](#checkpoint_cache_key)`checkpoint_cache_key` The key to use to store the snapshot position in `checkpoint_cache`. An alternative key can be provided if multiple CDC inputs share the same cache. **Type**: `string` **Default**: `microsoft_sql_server_cdc` ### [](#checkpoint_cache_table_name)`checkpoint_cache_table_name` The multipart identifier for the checkpoint cache table name. If no `checkpoint_cache` field is specified, this input will automatically create a table and stored procedure under the `rpcn` schema to act as a checkpoint cache. This table stores the latest processed Log Sequence Number (LSN) that has been successfully delivered, allowing Redpanda Connect to resume from that point upon restart rather than reconsume the entire change table. **Type**: `string` **Default**: `rpcn.CdcCheckpointCache` ```yaml # Examples: checkpoint_cache_table_name: dbo.checkpoint_cache ``` ### [](#checkpoint_limit)`checkpoint_limit` The maximum number of messages that can be processed concurrently before applying back pressure. Higher values enable better parallelization and batching but increase memory usage. Messages are processed in LSN order, and a given LSN is only acknowledged after all previous LSNs have been successfully delivered, ensuring at-least-once guarantees. **Type**: `int` **Default**: `1024` ### [](#connection_string)`connection_string` The connection string for the Microsoft SQL Server database. Use the format `sqlserver://username:password@host/instance?param1=value¶m2=value`. For Windows Authentication, use `sqlserver://host/instance?trusted_connection=yes`. Include additional parameters like `TrustServerCertificate=true` for self-signed certificates or `encrypt=disable` to disable encryption. **Type**: `string` ```yaml # Examples: connection_string: sqlserver://username:password@host/instance?param1=value¶m2=value ``` ### [](#exclude)`exclude[]` Regular expressions for tables to exclude from CDC streaming. Use this to filter out specific tables from the include patterns. Table names should follow the `schema.table` format. Exclude patterns are applied after include patterns, allowing you to include broad patterns while excluding specific tables. **Type**: `array` ```yaml # Examples: exclude: dbo.privatetable ``` ### [](#include)`include[]` Regular expressions for tables to include in CDC streaming. Specify table names using the format `schema.table` (such as `dbo.orders`, `sales.customers`). Each pattern is treated as a regular expression, allowing wildcards and pattern matching. All specified tables must have CDC enabled in SQL Server. **Type**: `array` ```yaml # Examples: include: dbo.products ``` ### [](#max_parallel_snapshot_tables)`max_parallel_snapshot_tables` Specifies a number of tables that will be processed in parallel during the snapshot processing stage. **Type**: `int` **Default**: `1` ### [](#snapshot_max_batch_size)`snapshot_max_batch_size` The maximum number of rows to stream in a single batch during the initial snapshot phase. Larger batch sizes can improve throughput for initial data loads but may increase memory usage. This setting only applies when `stream_snapshot` is enabled. **Type**: `int` **Default**: `1000` ### [](#stream_backoff_interval)`stream_backoff_interval` The time interval to wait between polling attempts when no new CDC data is available. For low-traffic tables, increasing this value reduces database load and network traffic. Use Go duration format like `5s`, `30s`, or `1m`. Shorter intervals provide lower latency for new changes but increase server load. **Type**: `string` **Default**: `5s` ```yaml # Examples: stream_backoff_interval: 5s # --- stream_backoff_interval: 1m ``` ### [](#stream_snapshot)`stream_snapshot` Whether to stream a snapshot of all existing data before streaming CDC changes. When enabled, the connector first queries all existing table data, then switches to streaming incremental changes from the transaction log. Set to `false` to start streaming only new changes from the current LSN position. **Type**: `bool` **Default**: `false` ```yaml # Examples: stream_snapshot: true ``` --- # Page 68: mongodb_cdc **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/mongodb_cdc.md --- # mongodb\_cdc --- title: mongodb_cdc latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/mongodb_cdc page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/mongodb_cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/mongodb_cdc.adoc page-git-created-date: "2025-03-11" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/mongodb_cdc/ "View the Cloud version of this component"), Self-Managed **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Streams data changes from a MongoDB replica set, using MongoDB’s [change streams](https://www.mongodb.com/docs/manual/changeStreams/) to capture data updates. Introduced in version 4.48.1. #### Common ```yml inputs: label: "" mongodb_cdc: url: "" # No default (required) database: "" # No default (required) username: "" password: "" collections: [] # No default (required) checkpoint_key: mongodb_cdc_checkpoint checkpoint_cache: "" # No default (required) checkpoint_interval: 5s checkpoint_limit: 1000 read_batch_size: 1000 read_max_wait: 1s stream_snapshot: false snapshot_parallelism: 1 auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" mongodb_cdc: url: "" # No default (required) database: "" # No default (required) username: "" password: "" collections: [] # No default (required) checkpoint_key: mongodb_cdc_checkpoint checkpoint_cache: "" # No default (required) checkpoint_interval: 5s checkpoint_limit: 1000 read_batch_size: 1000 read_max_wait: 1s stream_snapshot: false snapshot_parallelism: 1 snapshot_auto_bucket_sharding: false document_mode: update_lookup json_marshal_mode: canonical app_name: benthos auto_replay_nacks: true ``` ## [](#prerequisites)Prerequisites - MongoDB version 6 or later - Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. - A MongoDB database running as a [replica set](https://www.mongodb.com/docs/manual/replication/#replication-in-mongodb) or in a [sharded cluster](https://www.mongodb.com/docs/manual/sharding/) using replica set [protocol version 1](https://www.mongodb.com/docs/manual/reference/replica-configuration/#rsconf.protocolVersion). - A MongoDB database using the [WiredTiger](https://www.mongodb.com/docs/manual/core/wiredtiger/#storage-wiredtiger) storage engine. ## [](#enable-connectivity-from-cloud-based-data-sources)Enable connectivity from cloud-based data sources To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules. ## [](#data-capture-method)Data capture method The `mongodb_cdc` input uses [change streams](https://www.mongodb.com/docs/manual/changeStreams/) to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and [required collections](#collections). For more information, see [`document_mode` options](#document_mode) and the [MongoDB documentation](https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre—​and-post-images). ## [](#data-replication)Data replication Redpanda Connect allows you to specify which [database collections](#collections) in your source database to receive changes from. You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates. - Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded [operations log (oplog)](https://www.mongodb.com/docs/manual/core/replica-set-oplog/) position. - Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position. ### [](#snapshot-mode)Snapshot mode If you set the [`stream_snapshot` field](#stream_snapshot) to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections: 1. Records the latest oplog position. 2. Determines the strategy for splitting the snapshot data down into shards or chunks for more efficient processing: 1. If [`snapshot_auto_bucket_sharding`](#snapshot_auto_bucket_sharding) is set to `false`, the internal `$splitVector` command is used to compute shards. 2. If [`snapshot_auto_bucket_sharding`](#snapshot_auto_bucket_sharding) is set to `true`, the [`$bucketAuto`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/) command is used instead. This setting is for environments, such as MongoDB Atlas, where the `$splitVector` command is not available. 3. This input then uses the number of connections specified in [`snapshot-parallelism`](#snapshot_parallelism) to read the selected collections. > 📝 **NOTE** > > If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the [`checkpoint_cache`](#checkpoint_cache). 4. Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing. ### [](#streaming-mode)Streaming mode If you set the [`stream_snapshot` field](#stream_snapshot) to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the [`checkpoint_cache`](#checkpoint_cache). ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `operation`: The type of data change that generated the message: `read`, `create`, `update`, `replace`, `delete`, `update`. A `read` operation occurs when the initial snapshot of the database is processed. - `collection`: The name of the collection from which the message originated. - `operation_time`: The time the data change was written to the [operations log (oplog)](https://www.mongodb.com/docs/manual/core/replica-set-oplog/) in the form of a Binary JSON (BSON) timestamp: `{"t": , "i": }`. ## [](#fields)Fields ### [](#app_name)`app_name` The client application name. **Type**: `string` **Default**: `benthos` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#checkpoint_cache)`checkpoint_cache` Specify a [`cache` resource](../../caches/about/) to store the oplog position for the most recent data update streamed to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this position, avoiding the need to reprocess all collection updates. **Type**: `string` ### [](#checkpoint_interval)`checkpoint_interval` The interval between writing checkpoints to the cache. **Type**: `string` **Default**: `5s` ### [](#checkpoint_key)`checkpoint_key` The key identifier used to store the oplog position in [`checkpoint_cache`](#checkpoint_cache). If you have multiple `mongodb_cdc` inputs sharing the same cache, you can provide an alternative key. **Type**: `string` **Default**: `mongodb_cdc_checkpoint` ### [](#checkpoint_limit)`checkpoint_limit` The maximum number of in-flight messages emitted from this input. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given oplog position is not acknowledged until all messages under that offset are delivered. **Type**: `int` **Default**: `1000` ### [](#collections)`collections[]` A list of collections to stream changes from. Specify each collection name as a separate item. **Type**: `array` ### [](#database)`database` The name of the MongoDB database to stream changes from. **Type**: `string` ### [](#document_mode)`document_mode` The mode in which MongoDB emits document changes to Redpanda Connect, specifically updates and deletes. **Type**: `string` **Default**: `update_lookup` | Option | Summary | | --- | --- | | partial_update | In this mode update operations only have a description of the update operation, which follows the following schema: { "_id": , "operations": [ # type == set means that the value was updated like so: # root.foo."bar.baz" = "world" {"path": ["foo", "bar.baz"], "type": "set", "value":"world"}, # type == unset means that the value was deleted like so: # root.qux = deleted() {"path": ["qux"], "type": "unset", "value": null}, # type == truncatedArray means that the array at that path was truncated to value number of elements # root.array = this.array.slice(2) {"path": ["array"], "type": "truncatedArray", "value": 2} ] } | | pre_and_post_images | Uses pre and post image collection to emit the full documents for update and delete operations. To use and configure this mode see the setup steps in the ^MongoDB documentation. | | update_lookup | In this mode insert, replace and update operations have the full document emitted and deletes only have the _id field populated. Documents updates lookup the full document. This corresponds to the updateLookup option, see the ^MongoDB documentation for more information. | ### [](#json_marshal_mode)`json_marshal_mode` Controls the format used to convert a message from BSON to JSON when it is received by Redpanda Connect. **Type**: `string` **Default**: `canonical` | Option | Summary | | --- | --- | | canonical | A string format that emphasizes type preservation at the expense of readability and interoperability. That is, conversion from canonical to BSON will generally preserve type information except in certain specific cases. | | relaxed | A string format that emphasizes readability and interoperability at the expense of type preservation.That is, conversion from relaxed format to BSON can lose type information. | ### [](#password)`password` The password to connect to the database. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#read_batch_size)`read_batch_size` The number of documents to fetch in each message batch from MongoDB. **Type**: `int` **Default**: `1000` ### [](#read_max_wait)`read_max_wait` The maximum duration MongoDB waits to accumulate the [`read_batch_size`](#read_batch_size) documents on a change stream before returning the batch to Redpanda Connect. **Type**: `string` **Default**: `1s` ### [](#snapshot_auto_bucket_sharding)`snapshot_auto_bucket_sharding` Uses the [`$bucketAuto`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/) command instead of the default, `$splitVector`, to split the snapshot data into chunks for processing. This is required for environments, such as MongoDB Atlas, where the `$splitVector` command is not available. To enable parallel processing in these environments: - Set this field to to `true`. - Set `stream_snapshot` to `true`. - Increase `snapshot_parallelism` to a value greater than `1`. **Type**: `bool` **Default**: `false` ### [](#snapshot_parallelism)`snapshot_parallelism` Specifies the number of connections to use when reading the initial snapshot from one or more collections. Increase this number to enable parallel processing of the snapshot. This feature uses the `$splitVector` command to split snapshot data into chunks for more efficient processing. This field is only applicable when `stream_snapshot` is set to `true`. **Type**: `int` **Default**: `1` ### [](#stream_snapshot)`stream_snapshot` When set to `true`, this input streams a snapshot of all existing data in the source collections before streaming data changes. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target MongoDB server. **Type**: `string` ```yaml # Examples: url: mongodb://localhost:27017 ``` ### [](#username)`username` The username to connect to the database. **Type**: `string` **Default**: `""` --- # Page 69: mongodb **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/mongodb.md --- # mongodb --- title: mongodb latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/mongodb page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/mongodb.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/mongodb.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/mongodb/)[Cache](/redpanda-connect/components/caches/mongodb/)[Output](/redpanda-connect/components/outputs/mongodb/)[Processor](/redpanda-connect/components/processors/mongodb/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/mongodb/ "View the Cloud version of this component"), Self-Managed Executes a query and creates a message for each document received. Introduced in version 3.64.0. #### Common ```yml inputs: label: "" mongodb: url: "" # No default (required) database: "" # No default (required) username: "" password: "" collection: "" # No default (required) query: "" # No default (required) auto_replay_nacks: true batch_size: "" # No default (optional) sort: "" # No default (optional) limit: "" # No default (optional) ``` #### Advanced ```yml inputs: label: "" mongodb: url: "" # No default (required) database: "" # No default (required) username: "" password: "" app_name: benthos collection: "" # No default (required) operation: find json_marshal_mode: canonical query: "" # No default (required) auto_replay_nacks: true batch_size: "" # No default (optional) sort: "" # No default (optional) limit: "" # No default (optional) ``` Once the documents from the query are exhausted, this input shuts down, allowing the pipeline to gracefully terminate (or the next input in a [sequence](../sequence/) to execute). ## [](#fields)Fields ### [](#app_name)`app_name` The client application name. **Type**: `string` **Default**: `benthos` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#batch_size)`batch_size` A explicit number of documents to batch up before flushing them for processing. Must be greater than `0`. Operations: `find`, `aggregate` Requires version 4.26.0 or later. **Type**: `int` ```yaml # Examples: batch_size: 1000 ``` ### [](#collection)`collection` The collection to select from. **Type**: `string` ### [](#database)`database` The name of the target MongoDB database. **Type**: `string` ### [](#json_marshal_mode)`json_marshal_mode` The json\_marshal\_mode setting is optional and controls the format of the output message. Requires version 4.7.0 or later. **Type**: `string` **Default**: `canonical` | Option | Summary | | --- | --- | | canonical | A string format that emphasizes type preservation at the expense of readability and interoperability. That is, conversion from canonical to BSON will generally preserve type information except in certain specific cases. | | relaxed | A string format that emphasizes readability and interoperability at the expense of type preservation.That is, conversion from relaxed format to BSON can lose type information. | ### [](#limit)`limit` An explicit maximum number of documents to return. Operations: `find` Requires version 4.26.0 or later. **Type**: `int` ### [](#operation)`operation` The mongodb operation to perform. Requires version 4.2.0 or later. **Type**: `string` **Default**: `find` **Options**: `find`, `aggregate` ### [](#password)`password` The password to connect to the database. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#query)`query` Bloblang expression describing MongoDB query. **Type**: `string` ```yaml # Examples: query: |- root.from = {"$lte": timestamp_unix()} root.to = {"$gte": timestamp_unix()} ``` ### [](#sort)`sort` An object specifying fields to sort by, and the respective sort order (`1` ascending, `-1` descending). Note: The driver currently appears to support only one sorting key. Operations: `find` Requires version 4.26.0 or later. **Type**: `int` ```yaml # Examples: sort: name: 1 # --- sort: age: -1 ``` ### [](#url)`url` The URL of the target MongoDB server. **Type**: `string` ```yaml # Examples: url: mongodb://localhost:27017 ``` ### [](#username)`username` The username to connect to the database. **Type**: `string` **Default**: `""` --- # Page 70: mqtt **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/mqtt.md --- # mqtt --- title: mqtt latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/mqtt page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/mqtt.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/mqtt.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/mqtt/)[Output](/redpanda-connect/components/outputs/mqtt/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/mqtt/ "View the Cloud version of this component"), Self-Managed Subscribe to topics on MQTT brokers. Introduced in version 4.37.0. #### Common ```yml inputs: label: "" mqtt: urls: [] # No default (required) client_id: "" connect_timeout: 30s topics: [] # No default (required) auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" mqtt: urls: [] # No default (required) client_id: "" dynamic_client_id_suffix: "" # No default (optional) connect_timeout: 30s will: enabled: false qos: 0 retained: false topic: "" payload: "" user: "" password: "" keepalive: 30 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] topics: [] # No default (required) qos: 1 clean_session: true auto_replay_nacks: true ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: - mqtt\_duplicate - mqtt\_qos - mqtt\_retained - mqtt\_topic - mqtt\_message\_id You can access these metadata fields using [function interpolation](../../../configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#clean_session)`clean_session` Set whether the connection is non-persistent. **Type**: `bool` **Default**: `true` ### [](#client_id)`client_id` An identifier for the client connection. **Type**: `string` **Default**: `""` ### [](#connect_timeout)`connect_timeout` The maximum amount of time to wait in order to establish a connection before the attempt is abandoned. Requires version 3.58.0 or later. **Type**: `string` **Default**: `30s` ```yaml # Examples: connect_timeout: 1s # --- connect_timeout: 500ms ``` ### [](#dynamic_client_id_suffix)`dynamic_client_id_suffix` Append a dynamically generated suffix to the specified `client_id` on each run of the pipeline. This can be useful when clustering Redpanda Connect producers. **Type**: `string` | Option | Summary | | --- | --- | | nanoid | append a nanoid of length 21 characters | ### [](#keepalive)`keepalive` Max seconds of inactivity before a keepalive message is sent. **Type**: `int` **Default**: `30` ### [](#password)`password` A password to connect with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#qos)`qos` The level of delivery guarantee to enforce. Has options 0, 1, 2. **Type**: `int` **Default**: `1` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#topics)`topics[]` A list of topics to consume from. **Type**: `array` ### [](#urls)`urls[]` A list of URLs to connect to. Use the format `scheme://host:port`, where: - `scheme` is one of the following: `tcp`, `ssl`, `ws` - `host` is the IP address or hostname - `port` is the port on which the MQTT broker accepts connections If an item in the list contains commas, it is expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "tcp://localhost:1883" ``` ### [](#user)`user` A username to connect with. **Type**: `string` **Default**: `""` ### [](#will)`will` Set last will message in case of Redpanda Connect failure **Type**: `object` ### [](#will-enabled)`will.enabled` Whether to enable last will messages. **Type**: `bool` **Default**: `false` ### [](#will-payload)`will.payload` Set payload for last will message. **Type**: `string` **Default**: `""` ### [](#will-qos)`will.qos` Set QoS for last will message. Valid values are: 0, 1, 2. **Type**: `int` **Default**: `0` ### [](#will-retained)`will.retained` Set retained for last will message. **Type**: `bool` **Default**: `false` ### [](#will-topic)`will.topic` Set topic for last will message. **Type**: `string` **Default**: `""` --- # Page 71: mysql_cdc **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/mysql_cdc.md --- # mysql\_cdc --- title: mysql_cdc latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/mysql_cdc page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/mysql_cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/mysql_cdc.adoc page-git-created-date: "2025-02-20" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/mysql_cdc/ "View the Cloud version of this component"), Self-Managed **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Streams data changes from a MySQL database, using MySQL’s binary log to capture data updates. This input is built on the [`mysql-canal` library](https://github.com/go-mysql-org/go-mysql?tab=readme-ov-file#replication) but uses a custom approach for streaming historical data. Introduced in version 4.46.0. #### Common ```yml inputs: label: "" mysql_cdc: flavor: mysql dsn: "" # No default (required) tables: [] # No default (required) checkpoint_cache: "" # No default (required) checkpoint_key: mysql_binlog_position snapshot_max_batch_size: 1000 stream_snapshot: "" # No default (required) auto_replay_nacks: true checkpoint_limit: 1024 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml inputs: label: "" mysql_cdc: flavor: mysql dsn: "" # No default (required) tables: [] # No default (required) checkpoint_cache: "" # No default (required) checkpoint_key: mysql_binlog_position snapshot_max_batch_size: 1000 max_reconnect_attempts: 10 stream_snapshot: "" # No default (required) auto_replay_nacks: true checkpoint_limit: 1024 tls: skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] aws: enabled: false region: "" # No default (optional) endpoint: "" # No default (required) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) roles: [] # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` The `mysql_cdc` input uses MySQL’s [binary log (`binlog`)](https://dev.mysql.com/doc/refman/8.0/en/binary-log.html) to capture changes made to a MySQL database in real time and streams them to Redpanda Connect. Redpanda Connect allows you to specify which [database tables](#tables) in your source database to receive changes from. There are also [two replication modes](#choose-a-replication-mode) to choose from. ## [](#prerequisites)Prerequisites - MySQL version 8 or later - A MySQL instance with binary logging enabled ### [](#configuration-resources)Configuration resources #### Cloud platforms - [Change data capture on Amazon RDS for MySQL](https://aws.amazon.com/blogs/database/enable-change-data-capture-on-amazon-rds-for-mysql-applications-that-are-using-xa-transactions/) - [Azure MySQL Database (CDC)](https://learn.microsoft.com/en-us/fabric/real-time-hub/add-source-mysql-database-cdc) - [Google Cloud SQL for MySQL](https://cloud.google.com/datastream/docs/configure-cloudsql-mysql) #### Self-hosted MySQL - [Binary Logging Options and Variables](https://dev.mysql.com/doc/refman/8.4/en/replication-options-binary-log.html) ## [](#choose-a-replication-mode)Choose a replication mode You can run the `mysql_cdc` input in one of two modes, depending on whether you need a snapshot of existing data. - Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected tables and streams the contents before processing changes from the last recorded binlog position. - Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest binlog position. ### [](#snapshot-mode)Snapshot mode If you set the [`stream_snapshot` field](#stream_snapshot) to `true`, Redpanda Connect connects to your MySQL database and does the following to capture a snapshot of all data in the selected tables: 1. Executes the `FLUSH TABLES WITH READ LOCK` query to write any outstanding table updates to disk, and locks the tables. 2. Runs the `START TRANSACTION WITH CONSISTENT SNAPSHOT` statement to create a new transaction with a consistent view of all data, capturing the state of the database at the moment the transaction started. 3. Reads the current binlog position. 4. Runs the `UNLOCK TABLES` statement to release the database. 5. Preserves the initial transaction for data integrity. > 📝 **NOTE** > > If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current binlog position in the [`checkpoint_cache`](#checkpoint_cache). After the snapshot is taken, the input executes SELECT statements to extract data from the selected tables in two stages: 1. The input finds the primary keys of a table. 2. It selects the data ordered by primary key. Finally, the input uses the stored binlog position to catch up with changes that occurred during snapshot processing. ### [](#streaming-mode)Streaming mode If you set the [`stream_snapshot` field](#stream_snapshot) to `false`, Redpanda Connect connects to your MySQL database and starts processing data changes from the latest binlog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last binlog position written to the [`checkpoint_cache`](#checkpoint_cache). ## [](#binlog-rotation)Binlog rotation While the `mysql_cdc` input is streaming changes to Redpanda Connect, your MySQL server may rotate the binlog file. When this occurs, Redpanda Connect flushes the existing message batch and stores the new binlog position so that it can resume processing using the latest offset. ## [](#data-mappings)Data mappings The following table shows how selected MySQL data types are mapped to data types supported in Redpanda Connect. All other data types are mapped to string values. | MySQL data type | Bloblang value | | --- | --- | | TEXT, VARCHAR | A string value, for example: "this data" | | BINARY, VARBINARY, TINYBLOB, BLOB, MEDIUMBLOB, LONGBLOB | An array of byte values, for example: [byte1,byte2,byte3] | | DECIMAL, NUMERIC, TINYINT, SMALLINT, MEDIUMINT, INT, BIGINT, YEAR | A standard numeric type, for example: 123 | | FLOAT, DOUBLE | A 64-bit decimal (float64), for example: 123.1234 | | DATETIME, TIMESTAMP | A Bloblang timestamp, for example:1257894000000 2009-11-10 23:00:00 +0000 UTC | | SET | An array of strings, for example: ["apple", "banana", "orange"] | | JSON | A map object of the JSON, for example: {"red": 1, "blue": 2, "green": 3} | ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `operation`: The type of database operation that generated the message, such as `read`, `insert`, `update`, `delete`. A `read` operation occurs when a snapshot of the database is processed. - `table`: The name of the database table from which the message originated. - `binlog_position`: The [Binary Log (binlog)](https://dev.mysql.com/doc/refman/8.0/en/binary-log.html) position of each data update streamed from the source MySQL database. No `binlog_position` is set for data extracted from the initial snapshot. The `binlog` values are strings that you can sort to determine the order in which data updates occurred. ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#aws)`aws` AWS IAM authentication configuration for MySQL instances. When enabled, IAM credentials are used to generate temporary authentication tokens instead of a static password. **Type**: `object` ### [](#aws-enabled)`aws.enabled` Enable AWS IAM authentication for MySQL. When enabled, an IAM authentication token is generated and used as the password. When using IAM authentication ensure `max_reconnect_attempts` is set to a low value to ensure it can refresh credentials. **Type**: `bool` **Default**: `false` ### [](#aws-endpoint)`aws.endpoint` The MySQL endpoint hostname (e.g., mydb.abc123.us-east-1.rds.amazonaws.com). **Type**: `string` ### [](#aws-id)`aws.id` The ID of credentials to use. **Type**: `string` ### [](#aws-region)`aws.region` The AWS region where the MySQL instance is located. If no region is specified then the environment default will be used. **Type**: `string` ### [](#aws-role)`aws.role` Optional AWS IAM role ARN to assume for authentication. Alternatively, use `roles` array for role chaining instead. **Type**: `string` ### [](#aws-role_external_id)`aws.role_external_id` Optional external ID for the role assumption. Only used with the `role` field. Alternatively, use `roles` array for role chaining instead. **Type**: `string` ### [](#aws-roles)`aws.roles[]` Optional array of AWS IAM roles to assume for authentication. Roles can be assumed in sequence, enabling chaining for purposes such as cross-account access. Each role can optionally specify an external ID. **Type**: `object` ### [](#aws-roles-role)`aws.roles[].role` AWS IAM role ARN to assume. **Type**: `string` **Default**: `""` ### [](#aws-roles-role_external_id)`aws.roles[].role_external_id` Optional external ID for the role assumption. **Type**: `string` **Default**: `""` ### [](#aws-secret)`aws.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#aws-token)`aws.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#checkpoint_cache)`checkpoint_cache` Specify a `cache` resource to store the binlog position of the most recent data update delivered to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this last known position, avoiding the need to reprocess all table updates. **Type**: `string` ### [](#checkpoint_key)`checkpoint_key` The key identifier used to store the binlog position in [`checkpoint_cache`](#checkpoint_cache). If you have multiple `mysql_cdc` inputs sharing the same cache, you can provide an alternative key. **Type**: `string` **Default**: `mysql_binlog_position` ### [](#checkpoint_limit)`checkpoint_limit` The maximum number of messages that this input can process at a given time. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given binlog position is not acknowledged until all messages under that offset are delivered. **Type**: `int` **Default**: `1024` ### [](#dsn)`dsn` The data source name (DSN) of the MySQL database from which you want to stream updates. Use the format `user:password@tcp(localhost:3306)/database`. **Type**: `string` ```yaml # Examples: dsn: user:password@tcp(localhost:3306)/database ``` ### [](#flavor)`flavor` The type of MySQL database to connect to. **Type**: `string` **Default**: `mysql` | Option | Summary | | --- | --- | | mariadb | MariaDB flavored databases. | | mysql | MySQL flavored databases. | ### [](#max_reconnect_attempts)`max_reconnect_attempts` The maximum number of attempts the MySQL driver will try to re-establish a broken connection before Connect attempts reconnection. A zero or negative number means infinite retry attempts. **Type**: `int` **Default**: `10` ### [](#snapshot_max_batch_size)`snapshot_max_batch_size` The maximum number of table rows to fetch in each batch when taking a snapshot. This option is only available when `stream_snapshot` is set to `true`. **Type**: `int` **Default**: `1000` ### [](#stream_snapshot)`stream_snapshot` When set to `true`, this input streams a snapshot of all existing data in the source database before streaming data changes. To use this setting, all database tables that you want to replicate _must_ have a primary key. **Type**: `bool` ### [](#tables)`tables[]` A list of the database table names to stream changes from. Specify each table name as a separate item. **Type**: `array` ```yaml # Examples: tables: - table1 - table2 ``` ### [](#tls)`tls` Using this field overrides the SSL/TLS settings in the environment and DSN. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` --- # Page 72: nanomsg **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/nanomsg.md --- # nanomsg --- title: nanomsg latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/nanomsg page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/nanomsg.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/nanomsg.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/nanomsg/)[Output](/redpanda-connect/components/outputs/nanomsg/) **Available in:** Self-Managed Consumes messages via Nanomsg sockets (scalability protocols). #### Common ```yml inputs: label: "" nanomsg: urls: [] # No default (required) bind: true socket_type: PULL auto_replay_nacks: true sub_filters: [] ``` #### Advanced ```yml inputs: label: "" nanomsg: urls: [] # No default (required) bind: true socket_type: PULL auto_replay_nacks: true sub_filters: [] poll_timeout: 5s ``` Currently only PULL and SUB sockets are supported. ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#bind)`bind` Whether the URLs provided should be connected to, or bound as. **Type**: `bool` **Default**: `true` ### [](#poll_timeout)`poll_timeout` The period to wait until a poll is abandoned and reattempted. **Type**: `string` **Default**: `5s` ### [](#socket_type)`socket_type` The socket type to use. **Type**: `string` **Default**: `PULL` **Options**: `PULL`, `SUB` ### [](#sub_filters)`sub_filters[]` A list of subscription topic filters to use when consuming from a SUB socket. Specifying a single sub\_filter of `''` will subscribe to everything. **Type**: `array` **Default**: `[]` ### [](#urls)`urls[]` A list of URLs to connect to (or as). If an item of the list contains commas it will be expanded into multiple URLs. **Type**: `array` --- # Page 73: nats_jetstream **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/nats_jetstream.md --- # nats\_jetstream --- title: nats_jetstream latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/nats_jetstream page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/nats_jetstream.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/nats_jetstream.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/nats_jetstream/)[Output](/redpanda-connect/components/outputs/nats_jetstream/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/nats_jetstream/ "View the Cloud version of this component"), Self-Managed Reads messages from NATS JetStream subjects. Introduced in version 3.46.0. #### Common ```yml inputs: label: "" nats_jetstream: urls: [] # No default (required) queue: "" # No default (optional) subject: "" # No default (optional) durable: "" # No default (optional) stream: "" # No default (optional) bind: "" # No default (optional) deliver: all ``` #### Advanced ```yml inputs: label: "" nats_jetstream: urls: [] # No default (required) max_reconnects: "" # No default (optional) queue: "" # No default (optional) subject: "" # No default (optional) durable: "" # No default (optional) stream: "" # No default (optional) bind: "" # No default (optional) create_stream: false deliver: all ack_wait: 30s max_ack_pending: 1024 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) extract_tracing_map: "" # No default (optional) ``` ## [](#consume-mirrored-streams)Consume mirrored streams When a stream being consumed is mirrored in a different JetStream domain, the stream cannot be resolved from the subject name alone. You must specify the stream name as well as the subject (if applicable). ## [](#metadata)Metadata This input adds the following metadata fields to each message: ```text - nats_subject - nats_sequence_stream - nats_sequence_consumer - nats_num_delivered - nats_num_pending - nats_domain - nats_timestamp_unix_nano ``` You can access these metadata fields using [function interpolation](../../../configuration/interpolation/#bloblang-queries). ## [](#connection-name)Connection name When monitoring and managing a production [NATS system](https://docs.nats.io/nats-concepts/overview), it is often useful to know which connection a message was sent or received from. To achieve this, set the connection name option when creating a NATS connection. Redpanda Connect can then automatically set the connection name to the NATS component label, so that monitoring tools between NATS and Redpanda Connect can stay in sync. ## [](#authentication)Authentication A number of Redpanda Connect components use NATS services. Each of these components support optional, advanced authentication parameters for [NKeys](https://docs.nats.io/nats-server/configuration/securing_nats/auth_intro/nkey_auth) and [user credentials](https://docs.nats.io/using-nats/developer/connecting/creds). For an in-depth guide, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/nats_admin/security/jwt). ### [](#nkeys)NKeys NATS server can use NKeys in several ways for authentication. The simplest approach is to configure the server with a list of user’s public keys. The server can then generate a challenge for each connection request from a client, and the client must respond to the challenge by signing it with its private NKey, configured in the `nkey_file` or `nkey` field. For more details, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth). ### [](#user-credentials)User credentials NATS server also supports decentralized authentication based on JSON Web Tokens (JWTs). When a server is configured to use this authentication scheme, clients need a [user JWT](https://docs.nats.io/nats-server/configuration/securing_nats/jwt#json-web-tokens) and a corresponding [NKey secret](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth) to connect. You can use either of the following methods to supply the user JWT and NKey secret: - In the `user_credentials_file` field, enter the path to a file containing both the private key and the JWT. You can generate the file using the [nsc tool](https://docs.nats.io/nats-tools/nsc). - In the `user_jwt` field, enter a plain text JWT, and in the `user_nkey_seed` field, enter the plain text NKey seed or private key. For more details about authentication using JWTs, see the [NATS documentation](https://docs.nats.io/using-nats/developer/connecting/creds). ## [](#fields)Fields ### [](#ack_wait)`ack_wait` The maximum amount of time NATS server should wait for an ack from consumer. **Type**: `string` **Default**: `30s` ```yaml # Examples: ack_wait: 100ms # --- ack_wait: 5m ``` ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed or private key for NATS authentication. NKeys provide secure, cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#bind)`bind` Indicates that the subscription should use an existing consumer. **Type**: `bool` ### [](#create_stream)`create_stream` Whether to automatically create the stream if it doesn’t exist (requires the stream field to be set). **Type**: `bool` **Default**: `false` ### [](#deliver)`deliver` Determines which messages to deliver when consuming without a durable subscriber. **Type**: `string` **Default**: `all` | Option | Summary | | --- | --- | | all | Deliver all available messages. | | last | Deliver starting with the last published messages. | | last_per_subject | Deliver starting with the last published message per subject. | | new | Deliver starting from now, not taking into account any previous messages. | ### [](#durable)`durable` Preserve the state of your consumer under a durable name. **Type**: `string` ### [](#extract_tracing_map)`extract_tracing_map` EXPERIMENTAL: A [Bloblang mapping](../../../guides/bloblang/about/) that attempts to extract an object containing tracing propagation information, which will then be used as the root tracing span for the message. The specification of the extracted fields must match the format used by the service wide tracer. Requires version 4.23.0 or later. **Type**: `string` ```yaml # Examples: extract_tracing_map: root = @ # --- extract_tracing_map: root = this.meta.span ``` ### [](#max_ack_pending)`max_ack_pending` The maximum number of outstanding acks to be allowed before consuming is halted. **Type**: `int` **Default**: `1024` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. **Type**: `int` ### [](#queue)`queue` An optional queue group to consume as. **Type**: `string` ### [](#stream)`stream` A stream to consume from. Either a subject or stream must be specified. **Type**: `string` ### [](#subject)`subject` A subject to consume from. Supports wildcards for consuming multiple subjects. Either a subject or stream must be specified. **Type**: `string` ```yaml # Examples: subject: foo.bar.baz # --- subject: foo.*.baz # --- subject: foo.bar.* # --- subject: foo.> ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. Requires version 4.60.0 or later. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 74: nats_kv **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/nats_kv.md --- # nats\_kv --- title: nats_kv latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/nats_kv page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/nats_kv.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/nats_kv.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/nats_kv/)[Cache](/redpanda-connect/components/caches/nats_kv/)[Output](/redpanda-connect/components/outputs/nats_kv/)[Processor](/redpanda-connect/components/processors/nats_kv/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/nats_kv/ "View the Cloud version of this component"), Self-Managed Watches for updates in a NATS key-value bucket. Introduced in version 4.12.0. #### Common ```yml inputs: label: "" nats_kv: urls: [] # No default (required) bucket: "" # No default (required) key: > auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" nats_kv: urls: [] # No default (required) max_reconnects: "" # No default (optional) bucket: "" # No default (required) key: > auto_replay_nacks: true ignore_deletes: false include_history: false meta_only: false tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: ```text - nats_kv_key - nats_kv_bucket - nats_kv_revision - nats_kv_delta - nats_kv_operation - nats_kv_created ``` ## [](#connection-name)Connection name When monitoring and managing a production [NATS system](https://docs.nats.io/nats-concepts/overview), it is often useful to know which connection a message was sent or received from. To achieve this, set the connection name option when creating a NATS connection. Redpanda Connect can then automatically set the connection name to the NATS component label, so that monitoring tools between NATS and Redpanda Connect can stay in sync. ## [](#authentication)Authentication A number of Redpanda Connect components use NATS services. Each of these components support optional, advanced authentication parameters for [NKeys](https://docs.nats.io/nats-server/configuration/securing_nats/auth_intro/nkey_auth) and [user credentials](https://docs.nats.io/using-nats/developer/connecting/creds). For an in-depth guide, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/nats_admin/security/jwt). ### [](#nkeys)NKeys NATS server can use NKeys in several ways for authentication. The simplest approach is to configure the server with a list of user’s public keys. The server can then generate a challenge for each connection request from a client, and the client must respond to the challenge by signing it with its private NKey, configured in the `nkey_file` or `nkey` field. For more details, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth). ### [](#user-credentials)User credentials NATS server also supports decentralized authentication based on JSON Web Tokens (JWTs). When a server is configured to use this authentication scheme, clients need a [user JWT](https://docs.nats.io/nats-server/configuration/securing_nats/jwt#json-web-tokens) and a corresponding [NKey secret](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth) to connect. You can use either of the following methods to supply the user JWT and NKey secret: - In the `user_credentials_file` field, enter the path to a file containing both the private key and the JWT. You can generate the file using the [nsc tool](https://docs.nats.io/nats-tools/nsc). - In the `user_jwt` field, enter a plain text JWT, and in the `user_nkey_seed` field, enter the plain text NKey seed or private key. For more details about authentication using JWTs, see the [NATS documentation](https://docs.nats.io/using-nats/developer/connecting/creds). ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed or private key for NATS authentication. NKeys provide secure, cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the corresponding user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#bucket)`bucket` The name of the KV bucket. **Type**: `string` ```yaml # Examples: bucket: my_kv_bucket ``` ### [](#ignore_deletes)`ignore_deletes` Do not send delete markers as messages. **Type**: `bool` **Default**: `false` ### [](#include_history)`include_history` Include all the history per key, not just the last one. **Type**: `bool` **Default**: `false` ### [](#key)`key` Key to watch for updates, can include wildcards. **Type**: `string` **Default**: `>` ```yaml # Examples: key: foo.bar.baz # --- key: foo.*.baz # --- key: foo.bar.* # --- key: foo.> ``` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. **Type**: `int` ### [](#meta_only)`meta_only` Retrieve only the metadata of the entry **Type**: `bool` **Default**: `false` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. Requires version 4.60.0 or later. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 75: nats_stream **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/nats_stream.md --- # nats\_stream --- title: nats_stream latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/nats_stream page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/nats_stream.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/nats_stream.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/nats_stream/)[Output](/redpanda-connect/components/outputs/nats_stream/) **Available in:** Self-Managed Subscribe to a NATS Stream subject. Joining a queue is optional and allows multiple clients of a subject to consume using queue semantics. #### Common ```yml inputs: label: "" nats_stream: urls: [] # No default (required) cluster_id: "" # No default (required) client_id: "" queue: "" subject: "" durable_name: "" unsubscribe_on_close: false ``` #### Advanced ```yml inputs: label: "" nats_stream: urls: [] # No default (required) max_reconnects: "" # No default (optional) cluster_id: "" # No default (required) client_id: "" queue: "" subject: "" durable_name: "" unsubscribe_on_close: false start_from_oldest: true max_inflight: 1024 ack_wait: 30s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) extract_tracing_map: "" # No default (optional) ``` > ⚠️ **CAUTION: Deprecation notice** > > Deprecation notice > > The NATS Streaming Server is being deprecated. Critical bug fixes and security fixes will be applied until June of 2023. NATS-enabled applications requiring persistence should use [JetStream](https://docs.nats.io/nats-concepts/jetstream). Tracking and persisting offsets through a durable name is also optional and works with or without a queue. If a durable name is not provided then subjects are consumed from the most recently published message. When a consumer closes its connection, it unsubscribes from the subject. When all consumers of a durable queue do this, the offsets are deleted from the NATS server. To avoid this, set the `unsubscribe_on_close` field to `false`, to prevent consumers from unsubscribing. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - nats\_stream\_subject - nats\_stream\_sequence You can access these metadata fields using [function interpolation](../../../configuration/interpolation/#bloblang-queries). ## [](#authentication)Authentication There are a number of Redpanda Connect components that use NATS services. Each of these components support optional, advanced authentication parameters for [NKeys](https://docs.nats.io/nats-server/configuration/securing_nats/auth_intro/nkey_auth) and [user credentials](https://docs.nats.io/using-nats/developer/connecting/creds). For an in-depth guide, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/nats_admin/security/jwt). ### [](#nkeys)NKeys NATS server can use NKeys in several ways for authentication. The simplest approach is to configure the server with a list of user’s public keys. The server can then generate a challenge for each connection request from a client, and the client must respond to the challenge by signing it with its private NKey, configured in the `nkey_file` or `nkey` field. For more details, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth). ### [](#user-credentials)User credentials NATS server also supports decentralized authentication based on JSON Web Tokens (JWTs). When a server is configured to use this authentication scheme, clients need a [user JWT](https://docs.nats.io/nats-server/configuration/securing_nats/jwt#json-web-tokens) and a corresponding [NKey secret](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth) to connect. You can use either of the following methods to supply the user JWT and NKey secret: - In the `user_credentials_file` field, enter the path to a file containing both the private key and the JWT. You can generate the file using the [nsc tool](https://docs.nats.io/nats-tools/nsc). - In the `user_jwt` field, enter a plain text JWT, and in the `user_nkey_seed` field, enter the plain text NKey seed or private key. For more details about authentication using JWTs, see the [NATS documentation](https://docs.nats.io/using-nats/developer/connecting/creds). ## [](#fields)Fields ### [](#ack_wait)`ack_wait` An optional duration to specify at which a message that is yet to be acked will be automatically retried. **Type**: `string` **Default**: `30s` ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed or private key for NATS authentication. NKeys provide secure, cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the corresponding user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#client_id)`client_id` A client ID to connect as. **Type**: `string` **Default**: `""` ### [](#cluster_id)`cluster_id` The ID of the cluster to consume from. **Type**: `string` ### [](#durable_name)`durable_name` Preserve the state of your consumer under a durable name. **Type**: `string` **Default**: `""` ### [](#extract_tracing_map)`extract_tracing_map` EXPERIMENTAL: A [Bloblang mapping](../../../guides/bloblang/about/) that attempts to extract an object containing tracing propagation information, which is then used as the root tracing span for the message. The specification of the extracted fields must match the format used by the service wide tracer. Requires version 4.23.0 or later. **Type**: `string` ```yaml # Examples: extract_tracing_map: root = @ # --- extract_tracing_map: root = this.meta.span ``` ### [](#max_inflight)`max_inflight` The maximum number of unprocessed messages to fetch at a given time. **Type**: `int` **Default**: `1024` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. **Type**: `int` ### [](#queue)`queue` The queue to consume from. **Type**: `string` **Default**: `""` ### [](#start_from_oldest)`start_from_oldest` If a position is not found for a queue, determines whether to consume from the oldest available message, otherwise messages are consumed from the latest. **Type**: `bool` **Default**: `true` ### [](#subject)`subject` A subject to consume from. **Type**: `string` **Default**: `""` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. Requires version 4.60.0 or later. **Type**: `bool` **Default**: `false` ### [](#unsubscribe_on_close)`unsubscribe_on_close` Whether the subscription should be destroyed when this client disconnects. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 76: nats **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/nats.md --- # nats --- title: nats latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/nats page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/nats.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/nats.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/nats/)[Output](/redpanda-connect/components/outputs/nats/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/nats/ "View the Cloud version of this component"), Self-Managed Subscribe to a NATS subject. #### Common ```yml inputs: label: "" nats: urls: [] # No default (required) subject: "" # No default (required) queue: "" # No default (optional) auto_replay_nacks: true send_ack: true ``` #### Advanced ```yml inputs: label: "" nats: urls: [] # No default (required) max_reconnects: "" # No default (optional) subject: "" # No default (required) queue: "" # No default (optional) auto_replay_nacks: true send_ack: true nak_delay: "" # No default (optional) prefetch_count: 500000 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) extract_tracing_map: "" # No default (optional) ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: ```text - nats_subject - nats_reply_subject - All message headers (when supported by the connection) ``` You can access these metadata fields using [function interpolation](../../../configuration/interpolation/#bloblang-queries). ## [](#connection-name)Connection name When monitoring and managing a production [NATS system](https://docs.nats.io/nats-concepts/overview), it is often useful to know which connection a message was sent or received from. To achieve this, set the connection name option when creating a NATS connection. Redpanda Connect can then automatically set the connection name to the NATS component label, so that monitoring tools between NATS and Redpanda Connect can stay in sync. ## [](#authentication)Authentication A number of Redpanda Connect components use NATS services. Each of these components support optional, advanced authentication parameters for [NKeys](https://docs.nats.io/nats-server/configuration/securing_nats/auth_intro/nkey_auth) and [user credentials](https://docs.nats.io/using-nats/developer/connecting/creds). For an in-depth guide, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/nats_admin/security/jwt). ### [](#nkeys)NKeys NATS server can use NKeys in several ways for authentication. The simplest approach is to configure the server with a list of user’s public keys. The server can then generate a challenge for each connection request from a client, and the client must respond to the challenge by signing it with its private NKey, configured in the `nkey_file` or `nkey` field. For more details, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth). ### [](#user-credentials)User credentials NATS server also supports decentralized authentication based on JSON Web Tokens (JWTs). When a server is configured to use this authentication scheme, clients need a [user JWT](https://docs.nats.io/nats-server/configuration/securing_nats/jwt#json-web-tokens) and a corresponding [NKey secret](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth) to connect. You can use either of the following methods to supply the user JWT and NKey secret: - In the `user_credentials_file` field, enter the path to a file containing both the private key and the JWT. You can generate the file using the [nsc tool](https://docs.nats.io/nats-tools/nsc). - In the `user_jwt` field, enter a plain text JWT, and in the `user_nkey_seed` field, enter the plain text NKey seed or private key. For more details about authentication using JWTs, see the [NATS documentation](https://docs.nats.io/using-nats/developer/connecting/creds). ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed or private key for NATS authentication. NKeys provide secure, cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the corresponding user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#extract_tracing_map)`extract_tracing_map` EXPERIMENTAL: A [Bloblang mapping](../../../guides/bloblang/about/) that attempts to extract an object containing tracing propagation information, which will then be used as the root tracing span for the message. The specification of the extracted fields must match the format used by the service wide tracer. Requires version 4.23.0 or later. **Type**: `string` ```yaml # Examples: extract_tracing_map: root = @ # --- extract_tracing_map: root = this.meta.span ``` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. **Type**: `int` ### [](#nak_delay)`nak_delay` An optional delay duration on redelivering a message when negatively acknowledged. **Type**: `string` ```yaml # Examples: nak_delay: 1m ``` ### [](#prefetch_count)`prefetch_count` The maximum number of messages to pull at a time. **Type**: `int` **Default**: `500000` ### [](#queue)`queue` An optional queue group to consume as. **Type**: `string` ### [](#send_ack)`send_ack` Whether an automatic acknowledgment is sent as a reply to each message. When enabled, these replies are sent only when data has been delivered to all outputs. **Type**: `bool` **Default**: `true` ### [](#subject)`subject` A subject to consume from. Supports wildcards for consuming multiple subjects. Either a subject or stream must be specified. **Type**: `string` ```yaml # Examples: subject: foo.bar.baz # --- subject: foo.*.baz # --- subject: foo.bar.* # --- subject: foo.> ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. Requires version 4.60.0 or later. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 77: nsq **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/nsq.md --- # nsq --- title: nsq latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/nsq page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/nsq.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/nsq.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/nsq/)[Output](/redpanda-connect/components/outputs/nsq/) **Available in:** Self-Managed Subscribe to an NSQ instance topic and channel. #### Common ```yml inputs: label: "" nsq: nsqd_tcp_addresses: [] # No default (required) lookupd_http_addresses: [] # No default (required) topic: "" # No default (required) channel: "" # No default (required) user_agent: "" # No default (optional) max_in_flight: 100 max_attempts: 5 ``` #### Advanced ```yml inputs: label: "" nsq: nsqd_tcp_addresses: [] # No default (required) lookupd_http_addresses: [] # No default (required) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] topic: "" # No default (required) channel: "" # No default (required) user_agent: "" # No default (optional) max_in_flight: 100 max_attempts: 5 ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: - nsq\_attempts - nsq\_id - nsq\_nsqd\_address - nsq\_timestamp You can access these metadata fields using [function interpolation](../../../configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#channel)`channel` The channel to consume from. **Type**: `string` ### [](#lookupd_http_addresses)`lookupd_http_addresses[]` A list of nsqlookupd addresses to connect to. **Type**: `array` ### [](#max_attempts)`max_attempts` The maximum number of attempts to successfully consume a messages. **Type**: `int` **Default**: `5` ### [](#max_in_flight)`max_in_flight` The maximum number of pending messages to consume at any given time. **Type**: `int` **Default**: `100` ### [](#nsqd_tcp_addresses)`nsqd_tcp_addresses[]` A list of nsqd addresses to connect to. **Type**: `array` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#topic)`topic` The topic to consume from. **Type**: `string` ### [](#user_agent)`user_agent` A user agent to assume when connecting. **Type**: `string` --- # Page 78: ockam_kafka **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/ockam_kafka.md --- # ockam\_kafka --- title: ockam_kafka latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/ockam_kafka page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/ockam_kafka.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/ockam_kafka.adoc page-git-created-date: "2024-11-25" page-git-modified-date: "2025-10-06" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/ockam_kafka/)[Output](/redpanda-connect/components/outputs/ockam_kafka/) **Available in:** Self-Managed Uses [Ockam](https://docs.ockam.io/) to decrypt and read end-to-end encrypted messages from Kafka topics. You can write encrypted messages using the `ockam_kafka` output or by [creating a Kafka Portal Inlet](https://command.ockam.io/manual/ockam-kafka-inlet-create.html) using Ockam Command. [Ockam Secure Channels](https://docs.ockam.io/reference/command/secure-channels) guarantee that topic data can only be consumed by authenticated and authorized consumers, and that the data cannot be leaked or tampered with in-flight between the producer and consumer. Neither Kafka brokers, service providers nor other components can see or manipulate the messages. > 📝 **NOTE** > > You can use Ockam to encrypt whole messages, or specific fields in a message. Introduced in version 4.33.0. #### Common ```yml inputs: label: "" ockam_kafka: kafka: seed_brokers: [] # No default (optional) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) rack_id: "" instance_id: "" rebalance_timeout: 45s session_timeout: 1m heartbeat_interval: 3s start_offset: earliest fetch_max_bytes: 50MiB fetch_max_wait: 5s fetch_min_bytes: 1B fetch_max_partition_bytes: 1MiB transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) checkpoint_limit: 1024 commit_period: 5s multi_header: false batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) topic_lag_refresh_period: 5s disable_content_encryption: false enrollment_ticket: "" # No default (optional) identity_name: "" # No default (optional) allow: self route_to_kafka_outlet: self allow_producer: self relay: "" # No default (optional) node_address: 127.0.0.1:6262 encrypted_fields: [] ``` #### Advanced ```yml inputs: label: "" ockam_kafka: kafka: seed_brokers: [] # No default (optional) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) rack_id: "" instance_id: "" rebalance_timeout: 45s session_timeout: 1m heartbeat_interval: 3s start_offset: earliest fetch_max_bytes: 50MiB fetch_max_wait: 5s fetch_min_bytes: 1B fetch_max_partition_bytes: 1MiB transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) checkpoint_limit: 1024 commit_period: 5s multi_header: false batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) topic_lag_refresh_period: 5s disable_content_encryption: false enrollment_ticket: "" # No default (optional) identity_name: "" # No default (optional) allow: self route_to_kafka_outlet: self allow_producer: self relay: "" # No default (optional) node_address: 127.0.0.1:6262 encrypted_fields: [] ``` ## [](#fields)Fields ### [](#allow)`allow` Use in conjunction with the `route_to_kafka_outlet` field to specify an access control policy for the Kafka Portal Outlet. For example, setting this value to `kafka_us_east` forces the Kafka Outlet to present an Ockam credential, which confirms that the Outlet has the attribute `kafka_us_east=true`. **Type**: `string` **Default**: `self` ### [](#allow_producer)`allow_producer` Specify an access control policy for producers. For example, setting this value to `orders_producer` forces the producer to present an Ockam credential, which confirms that the producer has the attribute `orders_producer=true`. **Type**: `string` **Default**: `self` ### [](#disable_content_encryption)`disable_content_encryption` Disables Kafka message encryption. If this value is set to `true`: - Only message payloads remain unencrypted. This setting does not disable TLS or any other transport-layer encryption that may also be enabled. - All other `ockam_kafka` inlets and outlets in a topic must also have their settings set to `true`. **Type**: `bool` **Default**: `false` ### [](#encrypted_fields)`encrypted_fields[]` The fields to encrypt in the Kafka messages when the record is a valid JSON map. By default, the whole record is encrypted. **Type**: `array` **Default**: `[]` ### [](#enrollment_ticket)`enrollment_ticket` The path to a file or a URL where the enrollment ticket value is stored, or an inline hex-encoded value of the enrollment ticket (optional). You can generate a new ticket using the [`ockam project ticket` command](https://command.ockam.io/manual/ockam-project-ticket.html). **Type**: `string` ### [](#identity_name)`identity_name` The name of the [Ockam identity](https://command.ockam.io/manual/ockam-identity.html) to use. If this value is not provided, the default Ockam identity is automatically generated and used (optional). **Type**: `string` ### [](#kafka)`kafka` **Type**: `object` ### [](#kafka-batching)`kafka.batching` Configure a [batching policy](../../../configuration/batching/) for individual topic partitions. This allows the input to batch messages together before flushing them for processing. Batching may improve performance and is useful for windowed processing as it preserves the ordering of topic partitions. **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#kafka-batching-byte_size)`kafka.batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#kafka-batching-check)`kafka.batching.check` A [Bloblang query](../../../guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#kafka-batching-count)`kafka.batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#kafka-batching-period)`kafka.batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#kafka-batching-processors)`kafka.batching.processors[]` For aggregating and archiving message batches, you can add a list of [processors](../../processors/about/) to apply to a batch as it is flushed (optional). All resulting messages are flushed as a single batch even when you configure processors to split the batch into smaller batches. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#kafka-checkpoint_limit)`kafka.checkpoint_limit` The maximum number of messages that are processed in parallel inside the same partition before back pressure is applied. When a message with a specific offset is delivered to the output, the offset is only committed when all messages of previous offsets have also been delivered. This behavior ensures at-least-once delivery guarantees. However, in the event of crashes or server faults, it also increases the likelihood of duplicates. To decrease this risk, reduce the `checkpoint_limit` value. **Type**: `int` **Default**: `1024` ### [](#kafka-commit_period)`kafka.commit_period` The period of time between each commit of the current partition offsets. Offsets are always committed during shutdown. **Type**: `string` **Default**: `5s` ### [](#kafka-consumer_group)`kafka.consumer_group` Assign a consumer group for the processing of messages (optional). When this value is set: - Partitions of specified topics are automatically distributed across consumers sharing a consumer group. - Partition offsets are automatically committed and resumed under this name. Consumer groups are not supported when explicit partitions to consume from are specified in the `topics` field. **Type**: `string` ### [](#kafka-fetch_max_bytes)`kafka.fetch_max_bytes` The maximum size of a message batch (in bytes) that a broker tries to send during a client fetch. If individual records exceed the `fetch_max_bytes` value, brokers will still send them. **Type**: `string` **Default**: `50MiB` ### [](#kafka-fetch_max_partition_bytes)`kafka.fetch_max_partition_bytes` The maximum number of bytes that are consumed from a single partition in a fetch request. This field is equivalent to the Java setting `fetch.max.partition.bytes`. If a single batch is larger than the `fetch_max_partition_bytes` value, the batch is still sent so that the client can make progress. **Type**: `string` **Default**: `1MiB` ### [](#kafka-fetch_max_wait)`kafka.fetch_max_wait` The maximum period of time a broker can wait for a fetch response to reach the required minimum number of bytes (`fetch_min_bytes`). **Type**: `string` **Default**: `5s` ### [](#kafka-fetch_min_bytes)`kafka.fetch_min_bytes` The minimum number of bytes that a broker tries to send during a fetch. This field is equivalent to the Java setting `fetch.min.bytes`. **Type**: `string` **Default**: `1B` ### [](#kafka-heartbeat_interval)`kafka.heartbeat_interval` When you specify a `consumer_group`, `heartbeat_interval` sets how frequently a consumer group member should send heartbeats to Apache Kafka. Apache Kafka uses heartbeats to make sure that a group member’s session is active. You must set `heartbeat_interval` to less than one-third of `session_timeout`. This field is equivalent to the Java `heartbeat.interval.ms` setting and accepts Go duration format strings such as `10s` or `2m`. **Type**: `string` **Default**: `3s` ### [](#kafka-instance_id)`kafka.instance_id` When you specify a [`consumer_group`](#consumer_group), assign a unique value to `instance_id` to define the group’s static membership, which can prevent unnecessary rebalances during reconnections. When you assign an instance ID, the client does not automatically leave the consumer group when it disconnects. To remove the client, you must use an external admin command on behalf of the instance ID. **Type**: `string` **Default**: `""` ### [](#kafka-multi_header)`kafka.multi_header` Decode headers into lists to allow the handling of multiple values with the same key. **Type**: `bool` **Default**: `false` ### [](#kafka-rack_id)`kafka.rack_id` A rack identifier for this client. **Type**: `string` **Default**: `""` ### [](#kafka-rebalance_timeout)`kafka.rebalance_timeout` When you specify a [`consumer_group`](#consumer_group), `rebalance_timeout` sets a time limit for all consumer group members to complete their work and commit offsets after a rebalance has begun. The timeout excludes the time taken to detect a failed or late heartbeat, which indicates a rebalance is required. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `45s` ### [](#kafka-regexp_topics_exclude)`kafka.regexp_topics_exclude[]` A list of regular expression patterns for excluding topics when regex mode is enabled (via `regexp_topics` or `regexp_topics_include`). Topics matching any of these patterns will be excluded from consumption, even if they match include patterns. **Type**: `array` ### [](#kafka-regexp_topics_include)`kafka.regexp_topics_include[]` A list of regular expression patterns for matching topics to consume from. When specified, the client will periodically refresh the list of matching topics based on the `metadata_max_age` interval. This enables regex mode and cannot be used together with the `topics` field. Use `regexp_topics_exclude` to exclude specific patterns. **Type**: `array` ```yaml # Examples: regexp_topics_include: - logs_.* - metrics_.* # --- regexp_topics_include: - "events_[0-9]+" ``` ### [](#kafka-seed_brokers)`kafka.seed_brokers[]` A list of broker addresses to connect to (optional). List items that contain commas are expanded into multiple addresses. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" # --- seed_brokers: - "foo:9092" - "bar:9092" # --- seed_brokers: - "foo:9092,bar:9092" ``` ### [](#kafka-session_timeout)`kafka.session_timeout` When you specify a `consumer_group`, `session_timeout` sets the maximum interval between heartbeats sent by a consumer group member to the broker. If a broker doesn’t receive a heartbeat from a group member before the timeout expires, it removes the member from the consumer group and initiates a rebalance. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `1m` ### [](#kafka-start_offset)`kafka.start_offset` Specify the offset from which this input starts or restarts consuming messages. Restarts occur when the `OffsetOutOfRange` error is seen during a fetch. **Type**: `string` **Default**: `earliest` | Option | Summary | | --- | --- | | committed | Prevents consuming a partition in a group if the partition has no prior commits. Corresponds to Kafka’s auto.offset.reset=none option | | earliest | Start from the earliest offset. Corresponds to Kafka’s auto.offset.reset=earliest option. | | latest | Start from the latest offset. Corresponds to Kafka’s auto.offset.reset=latest option. | ### [](#kafka-tls)`kafka.tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#kafka-tls-client_certs)`kafka.tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#kafka-tls-client_certs-cert)`kafka.tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#kafka-tls-client_certs-cert_file)`kafka.tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#kafka-tls-client_certs-key)`kafka.tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#kafka-tls-client_certs-key_file)`kafka.tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#kafka-tls-client_certs-password)`kafka.tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#kafka-tls-enable_renegotiation)`kafka.tls.enable_renegotiation` Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#kafka-tls-enabled)`kafka.tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#kafka-tls-root_cas)`kafka.tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#kafka-tls-root_cas_file)`kafka.tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#kafka-tls-skip_cert_verify)`kafka.tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#kafka-topic_lag_refresh_period)`kafka.topic_lag_refresh_period` The interval between refresh cycles. During each cycle, this input queries the Redpanda Connect server to calculate the topic lag minus the number of produced messages that remain to be read from each topic/partition pair by the specified consumer group. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `5s` ### [](#kafka-topics)`kafka.topics[]` A list of topics to consume from (required). You can list multiple comma-separated topics in a single element. If you specify a `consumer_group`, partitions are automatically distributed across consumers of a topic. Otherwise, all partitions are consumed. Alternatively, add a colon after the topic name to set the explicit partitions to consume. For example, `foo:0` consumes the partition `0` of the topic `foo`. This syntax also supports ranges. For example, `foo:0-10` consumes all partitions from `0` through to `10` inclusively. Finally, add another colon after the partition to set an explicit offset to consume from. For example, `foo:0:10` consumes the partition `0` of the topic `foo` starting from the offset `10`. If the offset is not present (or remains unspecified) then the field `start_offset` determines which offset to start from. **Type**: `array` ```yaml # Examples: topics: - foo - bar # --- topics: - things.* # --- topics: - "foo,bar" # --- topics: - "foo:0" - "bar:1" - "bar:3" # --- topics: - "foo:0,bar:1,bar:3" # --- topics: - "foo:0-5" ``` ### [](#kafka-transaction_isolation_level)`kafka.transaction_isolation_level` The isolation level for handling transactional messages. This setting determines how transactions are processed and affects data consistency guarantees. **Type**: `string` **Default**: `read_uncommitted` | Option | Summary | | --- | --- | | read_committed | If set, only committed transactional records are processed. | | read_uncommitted | If set, then uncommitted records are processed. | ### [](#node_address)`node_address` The TCP listening address of the Ockam node. **Type**: `string` **Default**: `127.0.0.1:6262` ### [](#relay)`relay` Make the Ockam node accessible through a relay with the supplied name (optional). For example, setting this value to `orders_consumer` would require you to set the `route_to_consumer` on any producer to `/project/default/service/forward_to_orders_consumer/secure/api`. **Type**: `string` ### [](#route_to_kafka_outlet)`route_to_kafka_outlet` The route to reach the Kafka Portal Outlet of your Ockam portal. For example, `/project/default`. **Type**: `string` **Default**: `self` --- # Page 79: oracledb_cdc **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/oracledb_cdc.md --- # oracledb\_cdc --- title: oracledb_cdc latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/oracledb_cdc page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/oracledb_cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/oracledb_cdc.adoc description: Enables Change Data Capture by consuming from OracleDB. page-git-created-date: "2026-03-31" page-git-modified-date: "2026-03-31" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/oracledb_cdc/ "View the Cloud version of this component"), Self-Managed **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Enables Change Data Capture by consuming from OracleDB. Introduced in version 4.83.0. Streams changes from an Oracle database for Change Data Capture (CDC). Additionally, if `stream_snapshot` is set to true, existing data in the database is also streamed. #### Common ```yml inputs: label: "" oracledb_cdc: connection_string: "" # No default (required) wallet_path: "" # No default (optional) wallet_password: "" # No default (optional) stream_snapshot: false max_parallel_snapshot_tables: 1 snapshot_max_batch_size: 1000 logminer: scn_window_size: 20000 backoff_interval: 5s mining_interval: 300ms strategy: online_catalog max_transaction_events: 0 lob_enabled: true include: [] # No default (required) exclude: [] # No default (optional) checkpoint_cache: "" # No default (optional) checkpoint_cache_table_name: RPCN.CDC_CHECKPOINT_CACHE checkpoint_cache_key: oracledb_cdc checkpoint_limit: 1024 auto_replay_nacks: true batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml inputs: label: "" oracledb_cdc: connection_string: "" # No default (required) wallet_path: "" # No default (optional) wallet_password: "" # No default (optional) stream_snapshot: false max_parallel_snapshot_tables: 1 snapshot_max_batch_size: 1000 logminer: scn_window_size: 20000 backoff_interval: 5s mining_interval: 300ms strategy: online_catalog max_transaction_events: 0 lob_enabled: true include: [] # No default (required) exclude: [] # No default (optional) checkpoint_cache: "" # No default (optional) checkpoint_cache_table_name: RPCN.CDC_CHECKPOINT_CACHE checkpoint_cache_key: oracledb_cdc checkpoint_limit: 1024 auto_replay_nacks: true batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: - database\_schema: The database schema for the table where the message originates from. - table\_name: Name of the table that the message originated from. - operation: Type of operation that generated the message: "read", "delete", "insert", or "update". "read" is from messages that are read in the initial snapshot phase. - scn: The System Change Number in Oracle. - schema: The table schema, for use with schema-aware downstream processors such as `schema_registry_encode`. When new columns are detected in CDC events, the schema is automatically refreshed from the Oracle catalog. Dropped columns are reflected after a connector restart. ## [](#permissions)Permissions When using the default Oracle-based cache, the Connect user requires permission to create tables and stored procedures, and the rpcn schema must already exist. See `checkpoint_cache_table_name` for more information. ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#checkpoint_cache)`checkpoint_cache` A [cache resource](../../caches/about/) to use for storing the current System Change Number (SCN) that has been successfully delivered. This allows Redpanda Connect to continue from that SCN upon restart, rather than consume the entire state of OracleDB redo logs. If not set, the default Oracle-based cache is used. See `checkpoint_cache_table_name` for more information. **Type**: `string` ### [](#checkpoint_cache_key)`checkpoint_cache_key` The key to use to store the snapshot position in `checkpoint_cache`. An alternative key can be provided if multiple CDC inputs share the same cache. **Type**: `string` **Default**: `oracledb_cdc` ### [](#checkpoint_cache_table_name)`checkpoint_cache_table_name` The identifier for the checkpoint cache table name. If no `checkpoint_cache` field is specified, this input will automatically create a table and stored procedure under the `rpcn` schema to act as a checkpoint cache. This table stores the latest processed System Change Number (SCN) that has been successfully delivered, allowing Redpanda Connect to resume from that point upon restart rather than reconsume the entire redo log. **Type**: `string` **Default**: `RPCN.CDC_CHECKPOINT_CACHE` ```yaml # Examples: checkpoint_cache_table_name: RPCN.CHECKPOINT_CACHE ``` ### [](#checkpoint_limit)`checkpoint_limit` The maximum number of messages that can be processed at a given time. Increasing this limit enables parallel processing and batching at the output level. Any given System Change Number (SCN) will not be acknowledged unless all messages under that offset are delivered in order to preserve at least once delivery guarantees. **Type**: `int` **Default**: `1024` ### [](#connection_string)`connection_string` The connection string of the Oracle database to connect to. You can supply additional connection options as URL query parameters, for example: `oracle://user:password@host:1522/service?WALLET=/opt/oracle/wallet&SSL=true`. **Type**: `string` ```yaml # Examples: connection_string: oracle://username:password@host:port/service_name # --- connection_string: oracle://user:password@host:1522/service?WALLET=/opt/oracle/wallet&SSL=true ``` ### [](#exclude)`exclude[]` Regular expressions for tables to exclude. **Type**: `array` ```yaml # Examples: exclude: SCHEMA.PRIVATETABLE ``` ### [](#include)`include[]` Regular expressions for tables to include. **Type**: `array` ```yaml # Examples: include: SCHEMA.PRODUCTS ``` ### [](#logminer)`logminer` LogMiner configuration settings. **Type**: `object` ### [](#logminer-backoff_interval)`logminer.backoff_interval` The interval between attempts to check for new changes once all data is processed. For low traffic tables increasing this value can reduce network traffic to the server. **Type**: `string` **Default**: `5s` ```yaml # Examples: backoff_interval: 5s # --- backoff_interval: 1m ``` ### [](#logminer-lob_enabled)`logminer.lob_enabled` When enabled, large object (CLOB, BLOB) columns are included in both snapshot and streaming change events. When disabled, these columns are still present but contain no values. Enabling this option introduces additional performance overhead and increases memory requirements. **Type**: `bool` **Default**: `true` ### [](#logminer-max_transaction_events)`logminer.max_transaction_events` The maximum number of events that can be buffered for a single transaction. If a transaction exceeds this limit it is discarded and its events will not be emitted. Set to 0 to disable the limit. **Type**: `int` **Default**: `0` ### [](#logminer-mining_interval)`logminer.mining_interval` The interval between mining cycles during normal operation. Controls how frequently LogMiner polls for new changes when not caught up. **Type**: `string` **Default**: `300ms` ```yaml # Examples: mining_interval: 100ms # --- mining_interval: 1s ``` ### [](#logminer-scn_window_size)`logminer.scn_window_size` The SCN range to mine per cycle. Each cycle reads changes between the current SCN and current SCN + scn\_window\_size. Smaller values mean more frequent queries with lower memory usage but higher overhead; larger values reduce query frequency and improve throughput at the cost of higher memory usage per cycle. **Type**: `int` **Default**: `20000` ### [](#logminer-strategy)`logminer.strategy` Controls how LogMiner retrieves data dictionary information. `online_catalog` uses the current data dictionary for best performance but cannot capture DDL changes. Currently, only `online_catalog` is supported. **Type**: `string` **Default**: `online_catalog` ### [](#max_parallel_snapshot_tables)`max_parallel_snapshot_tables` Specifies a number of tables that will be processed in parallel during the snapshot processing stage. **Type**: `int` **Default**: `1` ### [](#snapshot_max_batch_size)`snapshot_max_batch_size` The maximum number of rows to be streamed in a single batch when taking a snapshot. **Type**: `int` **Default**: `1000` ### [](#stream_snapshot)`stream_snapshot` If set to true, the connector will query all the existing data as a part of snapshot process. Otherwise, it will start from the current System Change Number position. **Type**: `bool` **Default**: `false` ```yaml # Examples: stream_snapshot: true ``` ### [](#wallet_password)`wallet_password` Password for the `ewallet.p12` PKCS#12 wallet file. Only use this when the wallet directory contains `ewallet.p12` rather than `cwallet.sso`. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#wallet_path)`wallet_path` Path to the Oracle Wallet directory. When set, this automatically enables SSL. The directory must contain either `cwallet.sso` (auto-login, does not require a password) or `ewallet.p12` (requires `wallet_password`). **Type**: `string` ```yaml # Examples: wallet_path: /opt/oracle/wallet ``` --- # Page 80: otlp_grpc **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/otlp_grpc.md --- # otlp\_grpc --- title: otlp_grpc latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/otlp_grpc page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/otlp_grpc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/otlp_grpc.adoc description: Receive OpenTelemetry traces, logs, and metrics via OTLP/gRPC protocol. page-git-created-date: "2026-01-23" page-git-modified-date: "2026-01-23" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/otlp_grpc/)[Output](/redpanda-connect/components/outputs/otlp_grpc/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/otlp_grpc/ "View the Cloud version of this component"), Self-Managed **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Receive OpenTelemetry traces, logs, and metrics via OTLP/gRPC protocol. Introduced in version 4.78.0. Exposes an OpenTelemetry Collector gRPC receiver that accepts traces, logs, and metrics via gRPC. Telemetry data is received in OTLP protobuf format and converted to individual Redpanda OTEL v1 protobuf messages. Each signal (span, log record, or metric) becomes a separate message with embedded Resource and Scope metadata, optimized for Kafka partitioning. #### Common ```yml inputs: label: "" otlp_grpc: encoding: json address: 0.0.0.0:4317 rate_limit: "" ``` #### Advanced ```yml inputs: label: "" otlp_grpc: encoding: json address: 0.0.0.0:4317 tls: enabled: false cert_file: "" key_file: "" auth_token: "" max_recv_msg_size: 4194304 rate_limit: "" tcp: reuse_addr: false reuse_port: false schema_registry: url: "" # No default (required) timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth2: enabled: false client_key: "" client_secret: "" token_url: "" scopes: [] endpoint_params: {} oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} common_subject: "" trace_subject: "" log_subject: "" metric_subject: "" ``` ## [](#protocols)Protocols This input supports OTLP/gRPC on the default port 4317 using the standard OTLP protobuf format for all signal types (traces, logs, metrics). ## [](#output-format)Output format Each OTLP export request is unbatched into individual messages: - **Traces**: One message per span - **Logs**: One message per log record - **Metrics**: One message per metric Messages are encoded in Redpanda OTEL v1 protobuf format. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `signal_type` - The signal type: "trace", "log", or "metric" You can access these metadata fields using [function interpolation](../../../configuration/interpolation/#bloblang-queries). ## [](#authentication)Authentication When `auth_token` is configured, clients must include the token in the gRPC metadata. ### [](#go-client-example)Go client example ```go import ( "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc" ) exporter, err := otlptracegrpc.New(ctx, otlptracegrpc.WithEndpoint("localhost:4317"), otlptracegrpc.WithInsecure(), // or WithTLSCredentials() for TLS otlptracegrpc.WithHeaders(map[string]string{ "authorization": "Bearer your-token-here", }), ) ``` ### [](#environment-variable)Environment variable ```bash export OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer your-token-here" ``` ## [](#rate-limiting)Rate limiting An optional rate limit resource can be specified to throttle incoming requests. When the rate limit is breached, requests will receive a ResourceExhausted gRPC status code. ## [](#fields)Fields ### [](#address)`address` The address to listen on for gRPC connections. **Type**: `string` **Default**: `0.0.0.0:4317` ### [](#auth_token)`auth_token` Optional bearer token for authentication. When set, requests must include 'authorization: Bearer ' metadata. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#encoding)`encoding` Encoding format for messages in the batch. Options: 'protobuf' or 'json'. **Type**: `string` **Default**: `json` **Options**: `protobuf`, `json` ### [](#max_recv_msg_size)`max_recv_msg_size` Maximum size of gRPC messages to receive in bytes. **Type**: `int` **Default**: `4194304` ### [](#rate_limit)`rate_limit` An optional rate limit resource to throttle requests. **Type**: `string` **Default**: `""` ### [](#schema_registry)`schema_registry` Optional Schema Registry configuration for adding Schema Registry wire format headers to messages. **Type**: `object` ### [](#schema_registry-basic_auth)`schema_registry.basic_auth` Allows you to specify basic authentication. **Type**: `object` ### [](#schema_registry-basic_auth-enabled)`schema_registry.basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-basic_auth-password)`schema_registry.basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-basic_auth-username)`schema_registry.basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#schema_registry-common_subject)`schema_registry.common_subject` Schema subject name for the common protobuf schema. Only used when encoding is 'protobuf'. Defaults to 'redpanda-otel-common' for protobuf encoding or 'redpanda-otel-common-json' for JSON encoding. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt)`schema_registry.jwt` Beta Allows you to specify JWT authentication. **Type**: `object` ### [](#schema_registry-jwt-claims)`schema_registry.jwt.claims` A value used to identify the claims that issued the JWT. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-enabled)`schema_registry.jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-jwt-headers)`schema_registry.jwt.headers` Add optional key/value headers to the JWT. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-private_key_file)`schema_registry.jwt.private_key_file` A file with the PEM encoded via PKCS1 or PKCS8 as private key. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt-signing_method)`schema_registry.jwt.signing_method` A method used to sign the token such as RS256, RS384, RS512 or EdDSA. **Type**: `string` **Default**: `""` ### [](#schema_registry-log_subject)`schema_registry.log_subject` Schema subject name for log data. Defaults to 'redpanda-otel-logs' for protobuf encoding or 'redpanda-otel-logs-json' for JSON encoding. **Type**: `string` **Default**: `""` ### [](#schema_registry-metric_subject)`schema_registry.metric_subject` Schema subject name for metric data. Defaults to 'redpanda-otel-metrics' for protobuf encoding or 'redpanda-otel-metrics-json' for JSON encoding. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth)`schema_registry.oauth` Allows you to specify open authentication via OAuth version 1. **Type**: `object` ### [](#schema_registry-oauth-access_token)`schema_registry.oauth.access_token` A value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-access_token_secret)`schema_registry.oauth.access_token_secret` A secret provided in order to establish ownership of a given access token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_key)`schema_registry.oauth.consumer_key` A value used to identify the client to the service provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_secret)`schema_registry.oauth.consumer_secret` A secret used to establish ownership of the consumer key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-enabled)`schema_registry.oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-oauth2)`schema_registry.oauth2` Allows you to specify open authentication via OAuth version 2 using the client credentials token flow. **Type**: `object` ### [](#schema_registry-oauth2-client_key)`schema_registry.oauth2.client_key` A value used to identify the client to the token provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth2-client_secret)`schema_registry.oauth2.client_secret` A secret used to establish ownership of the client key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth2-enabled)`schema_registry.oauth2.enabled` Whether to use OAuth version 2 in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-oauth2-endpoint_params)`schema_registry.oauth2.endpoint_params` A list of optional endpoint parameters, values should be arrays of strings. **Type**: `object` **Default**: `{}` ```yaml # Examples: endpoint_params: audience: - https://example.com resource: - https://api.example.com ``` ### [](#schema_registry-oauth2-scopes)`schema_registry.oauth2.scopes[]` A list of optional requested permissions. **Type**: `array` **Default**: `[]` ### [](#schema_registry-oauth2-token_url)`schema_registry.oauth2.token_url` The URL of the token provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-timeout)`schema_registry.timeout` HTTP client timeout for Schema Registry requests. **Type**: `string` **Default**: `5s` ### [](#schema_registry-tls)`schema_registry.tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#schema_registry-tls-client_certs)`schema_registry.tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#schema_registry-tls-client_certs-cert)`schema_registry.tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-cert_file)`schema_registry.tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key)`schema_registry.tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key_file)`schema_registry.tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-password)`schema_registry.tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#schema_registry-tls-enable_renegotiation)`schema_registry.tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-enabled)`schema_registry.tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-root_cas)`schema_registry.tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#schema_registry-tls-root_cas_file)`schema_registry.tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#schema_registry-tls-skip_cert_verify)`schema_registry.tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#schema_registry-trace_subject)`schema_registry.trace_subject` Schema subject name for trace data. Defaults to 'redpanda-otel-traces' for protobuf encoding or 'redpanda-otel-traces-json' for JSON encoding. **Type**: `string` **Default**: `""` ### [](#schema_registry-url)`schema_registry.url` Schema Registry URL for schema operations. **Type**: `string` ```yaml # Examples: url: http://localhost:8081 ``` ### [](#tcp)`tcp` TCP listener socket configuration. **Type**: `object` ### [](#tcp-reuse_addr)`tcp.reuse_addr` Enable SO\_REUSEADDR, allowing binding to ports in TIME\_WAIT state. Useful for graceful restarts and config reloads where the server needs to rebind to the same port immediately after shutdown. **Type**: `bool` **Default**: `false` ### [](#tcp-reuse_port)`tcp.reuse_port` Enable SO\_REUSEPORT, allowing multiple sockets to bind to the same port for load balancing across multiple processes/threads. **Type**: `bool` **Default**: `false` ### [](#tls)`tls` TLS configuration for gRPC. **Type**: `object` ### [](#tls-cert_file)`tls.cert_file` Path to the TLS certificate file. **Type**: `string` **Default**: `""` ### [](#tls-enabled)`tls.enabled` Enable TLS connections. **Type**: `bool` **Default**: `false` ### [](#tls-key_file)`tls.key_file` Path to the TLS key file. **Type**: `string` **Default**: `""` --- # Page 81: otlp_http **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/otlp_http.md --- # otlp\_http --- title: otlp_http latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/otlp_http page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/otlp_http.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/otlp_http.adoc description: Receive OpenTelemetry traces, logs, and metrics via OTLP/HTTP protocol. page-git-created-date: "2026-01-23" page-git-modified-date: "2026-01-23" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/otlp_http/)[Output](/redpanda-connect/components/outputs/otlp_http/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/otlp_http/ "View the Cloud version of this component"), Self-Managed **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Receive OpenTelemetry traces, logs, and metrics via OTLP/HTTP protocol. Introduced in version 4.78.0. Exposes an OpenTelemetry Collector HTTP receiver that accepts traces, logs, and metrics via HTTP. Telemetry data is received in OTLP format (both protobuf and JSON) at standard OTLP endpoints and converted to individual Redpanda OTEL v1 protobuf messages. Each signal (span, log record, or metric) becomes a separate message with embedded Resource and Scope metadata, optimized for Kafka partitioning. #### Common ```yml inputs: label: "" otlp_http: encoding: json address: 0.0.0.0:4318 rate_limit: "" ``` #### Advanced ```yml inputs: label: "" otlp_http: encoding: json address: 0.0.0.0:4318 tls: enabled: false cert_file: "" key_file: "" auth_token: "" read_timeout: 10s write_timeout: 10s max_body_size: 4194304 rate_limit: "" tcp: reuse_addr: false reuse_port: false schema_registry: url: "" # No default (required) timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth2: enabled: false client_key: "" client_secret: "" token_url: "" scopes: [] endpoint_params: {} oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} common_subject: "" trace_subject: "" log_subject: "" metric_subject: "" ``` ## [](#endpoints)Endpoints This input exposes the following standard OTLP HTTP endpoints: - `/v1/traces` - OpenTelemetry traces - `/v1/logs` - OpenTelemetry logs - `/v1/metrics` - OpenTelemetry metrics ## [](#protocols)Protocols This input supports OTLP/HTTP on the default port 4318. It accepts both: - `application/x-protobuf` - OTLP protobuf format - `application/json` - OTLP JSON format ## [](#output-format)Output format Each OTLP export request is unbatched into individual messages: - **Traces**: One message per span - **Logs**: One message per log record - **Metrics**: One message per metric Messages are encoded in Redpanda OTEL v1 protobuf format. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `signal_type` - The signal type: "trace", "log", or "metric" You can access these metadata fields using [function interpolation](../../../configuration/interpolation/#bloblang-queries). ## [](#authentication)Authentication When `auth_token` is configured, clients must include the token in the HTTP Authorization header. ### [](#go-client-example)Go client example ```go import ( "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" ) exporter, err := otlptracehttp.New(ctx, otlptracehttp.WithEndpoint("localhost:4318"), otlptracehttp.WithInsecure(), // or WithTLSClientConfig() for TLS otlptracehttp.WithHeaders(map[string]string{ "Authorization": "Bearer your-token-here", }), ) ``` ### [](#curl-example)cURL example ```bash curl -X POST http://localhost:4318/v1/traces \ -H "Content-Type: application/x-protobuf" \ -H "Authorization: Bearer your-token-here" \ --data-binary @traces.pb ``` ### [](#environment-variable)Environment variable ```bash export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer your-token-here" ``` ## [](#rate-limiting)Rate limiting An optional rate limit resource can be specified to throttle incoming requests. When the rate limit is breached, requests will receive a 429 (Too Many Requests) response. ## [](#fields)Fields ### [](#address)`address` The address to listen on for HTTP connections. **Type**: `string` **Default**: `0.0.0.0:4318` ### [](#auth_token)`auth_token` Optional bearer token for authentication. When set, requests must include 'Authorization: Bearer ' header. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#encoding)`encoding` Encoding format for messages in the batch. Options: 'protobuf' or 'json'. **Type**: `string` **Default**: `json` **Options**: `protobuf`, `json` ### [](#max_body_size)`max_body_size` Maximum size of HTTP request body in bytes. **Type**: `int` **Default**: `4194304` ### [](#rate_limit)`rate_limit` An optional rate limit resource to throttle requests. **Type**: `string` **Default**: `""` ### [](#read_timeout)`read_timeout` Maximum duration for reading the entire request. **Type**: `string` **Default**: `10s` ### [](#schema_registry)`schema_registry` Optional Schema Registry configuration for adding Schema Registry wire format headers to messages. **Type**: `object` ### [](#schema_registry-basic_auth)`schema_registry.basic_auth` Allows you to specify basic authentication. **Type**: `object` ### [](#schema_registry-basic_auth-enabled)`schema_registry.basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-basic_auth-password)`schema_registry.basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-basic_auth-username)`schema_registry.basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#schema_registry-common_subject)`schema_registry.common_subject` Schema subject name for the common protobuf schema. Only used when encoding is 'protobuf'. Defaults to 'redpanda-otel-common' for protobuf encoding or 'redpanda-otel-common-json' for JSON encoding. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt)`schema_registry.jwt` Beta Allows you to specify JWT authentication. **Type**: `object` ### [](#schema_registry-jwt-claims)`schema_registry.jwt.claims` A value used to identify the claims that issued the JWT. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-enabled)`schema_registry.jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-jwt-headers)`schema_registry.jwt.headers` Add optional key/value headers to the JWT. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-private_key_file)`schema_registry.jwt.private_key_file` A file with the PEM encoded via PKCS1 or PKCS8 as private key. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt-signing_method)`schema_registry.jwt.signing_method` A method used to sign the token such as RS256, RS384, RS512 or EdDSA. **Type**: `string` **Default**: `""` ### [](#schema_registry-log_subject)`schema_registry.log_subject` Schema subject name for log data. Defaults to 'redpanda-otel-logs' for protobuf encoding or 'redpanda-otel-logs-json' for JSON encoding. **Type**: `string` **Default**: `""` ### [](#schema_registry-metric_subject)`schema_registry.metric_subject` Schema subject name for metric data. Defaults to 'redpanda-otel-metrics' for protobuf encoding or 'redpanda-otel-metrics-json' for JSON encoding. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth)`schema_registry.oauth` Allows you to specify open authentication via OAuth version 1. **Type**: `object` ### [](#schema_registry-oauth-access_token)`schema_registry.oauth.access_token` A value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-access_token_secret)`schema_registry.oauth.access_token_secret` A secret provided in order to establish ownership of a given access token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_key)`schema_registry.oauth.consumer_key` A value used to identify the client to the service provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_secret)`schema_registry.oauth.consumer_secret` A secret used to establish ownership of the consumer key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-enabled)`schema_registry.oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-oauth2)`schema_registry.oauth2` Allows you to specify open authentication via OAuth version 2 using the client credentials token flow. **Type**: `object` ### [](#schema_registry-oauth2-client_key)`schema_registry.oauth2.client_key` A value used to identify the client to the token provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth2-client_secret)`schema_registry.oauth2.client_secret` A secret used to establish ownership of the client key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth2-enabled)`schema_registry.oauth2.enabled` Whether to use OAuth version 2 in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-oauth2-endpoint_params)`schema_registry.oauth2.endpoint_params` A list of optional endpoint parameters, values should be arrays of strings. **Type**: `object` **Default**: `{}` ```yaml # Examples: endpoint_params: audience: - https://example.com resource: - https://api.example.com ``` ### [](#schema_registry-oauth2-scopes)`schema_registry.oauth2.scopes[]` A list of optional requested permissions. **Type**: `array` **Default**: `[]` ### [](#schema_registry-oauth2-token_url)`schema_registry.oauth2.token_url` The URL of the token provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-timeout)`schema_registry.timeout` HTTP client timeout for Schema Registry requests. **Type**: `string` **Default**: `5s` ### [](#schema_registry-tls)`schema_registry.tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#schema_registry-tls-client_certs)`schema_registry.tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#schema_registry-tls-client_certs-cert)`schema_registry.tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-cert_file)`schema_registry.tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key)`schema_registry.tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key_file)`schema_registry.tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-password)`schema_registry.tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#schema_registry-tls-enable_renegotiation)`schema_registry.tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-enabled)`schema_registry.tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-root_cas)`schema_registry.tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#schema_registry-tls-root_cas_file)`schema_registry.tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#schema_registry-tls-skip_cert_verify)`schema_registry.tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#schema_registry-trace_subject)`schema_registry.trace_subject` Schema subject name for trace data. Defaults to 'redpanda-otel-traces' for protobuf encoding or 'redpanda-otel-traces-json' for JSON encoding. **Type**: `string` **Default**: `""` ### [](#schema_registry-url)`schema_registry.url` Schema Registry URL for schema operations. **Type**: `string` ```yaml # Examples: url: http://localhost:8081 ``` ### [](#tcp)`tcp` TCP listener socket configuration. **Type**: `object` ### [](#tcp-reuse_addr)`tcp.reuse_addr` Enable SO\_REUSEADDR, allowing binding to ports in TIME\_WAIT state. Useful for graceful restarts and config reloads where the server needs to rebind to the same port immediately after shutdown. **Type**: `bool` **Default**: `false` ### [](#tcp-reuse_port)`tcp.reuse_port` Enable SO\_REUSEPORT, allowing multiple sockets to bind to the same port for load balancing across multiple processes/threads. **Type**: `bool` **Default**: `false` ### [](#tls)`tls` TLS configuration for HTTP. **Type**: `object` ### [](#tls-cert_file)`tls.cert_file` Path to the TLS certificate file. **Type**: `string` **Default**: `""` ### [](#tls-enabled)`tls.enabled` Enable TLS connections. **Type**: `bool` **Default**: `false` ### [](#tls-key_file)`tls.key_file` Path to the TLS key file. **Type**: `string` **Default**: `""` ### [](#write_timeout)`write_timeout` Maximum duration for writing the response. **Type**: `string` **Default**: `10s` --- # Page 82: parquet **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/parquet.md --- # parquet --- title: parquet latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/parquet page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/parquet.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/parquet.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/parquet/)[Processor](/redpanda-connect/components/processors/parquet/) **Available in:** Self-Managed Reads and decodes [Parquet files](https://parquet.apache.org/docs/) into a stream of structured messages. Introduced in version 4.8.0. #### Common ```yml inputs: label: "" parquet: paths: [] # No default (required) auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" parquet: paths: [] # No default (required) batch_count: 1 auto_replay_nacks: true ``` This input uses [https://github.com/parquet-go/parquet-go](https://github.com/parquet-go/parquet-go), which is itself experimental. Therefore changes could be made into how this processor functions outside of major version releases. By default any BYTE\_ARRAY or FIXED\_LEN\_BYTE\_ARRAY value will be extracted as a byte slice (`[]byte`) unless the logical type is UTF8, in which case they are extracted as a string (`string`). When a value extracted as a byte slice exists within a document which is later JSON serialized by default it will be base 64 encoded into strings, which is the default for arbitrary data fields. It is possible to convert these binary values to strings (or other data types) using Bloblang transformations such as `root.foo = this.foo.string()` or `root.foo = this.foo.encode("hex")`, etc. ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#batch_count)`batch_count` Optionally process records in batches. This can help to speed up the consumption of exceptionally large files. When the end of the file is reached the remaining records are processed as a (potentially smaller) batch. **Type**: `int` **Default**: `1` ### [](#paths)`paths[]` A list of file paths to read from. Each file will be read sequentially until the list is exhausted, at which point the input will close. Glob patterns are supported, including super globs (double star). **Type**: `array` ```yaml # Examples: paths: /tmp/foo.parquet # --- paths: /tmp/bar/*.parquet # --- paths: /tmp/data/**/*.parquet ``` --- # Page 83: postgres_cdc **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/postgres_cdc.md --- # postgres\_cdc --- title: postgres_cdc latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/postgres_cdc page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/postgres_cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/postgres_cdc.adoc page-git-created-date: "2024-12-05" page-git-modified-date: "2025-10-07" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/postgres_cdc/ "View the Cloud version of this component"), Self-Managed **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Streams data changes from a PostgreSQL database using logical replication. There is also a configuration option to [stream all existing data](#stream_snapshot) from the database. Introduced in version 4.40.0 and renamed from `pg_stream` to `postgres_cdc` in version 4.43.0. ```yml inputs: label: "" postgres_cdc: dsn: "" # No default (required) include_transaction_markers: false stream_snapshot: false snapshot_batch_size: 1000 schema: "" # No default (required) tables: [] # No default (required) checkpoint_limit: 1024 temporary_slot: false slot_name: "" # No default (required) pg_standby_timeout: 10s pg_wal_monitor_interval: 3s max_parallel_snapshot_tables: 1 auto_replay_nacks: true batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` The `postgres_cdc` input uses logical replication to capture changes made to a PostgreSQL database in real time and streams them to Redpanda Connect. Redpanda Connect uses this replication method to allow you to choose which database tables in your source database to receive changes from. There are also [two replication modes](#choose-a-replication-mode) to choose from, and an [option to receive TOAST and deleted values](#receive-toast-and-deleted-values) in your data updates. ## [](#prerequisites)Prerequisites - PostgreSQL version 14 or later - Logical replication enabled on your PostgreSQL cluster To check whether logical replication is already enabled, run the following query: ```SQL SHOW wal_level; ``` If the `wal_level` value is `logical`, you can start to use this connector. Otherwise, choose from the following sets of instructions to update your replication settings. ### Cloud platforms - [Amazon RDS for PostgreSQL DB](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/PostgreSQL.Concepts.General.FeatureSupport.LogicalReplication.html) - [Azure Database for PostgreSQL](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-logical#prerequisites-for-logical-replication-and-logical-decoding) - [Google Cloud SQL for PostgreSQL](https://cloud.google.com/sql/docs/postgres/replication/configure-logical-replication), including creating a user with replication privileges - [Neon](https://neon.tech/docs/guides/logical-replication-guide) ### Self-Hosted PostgreSQL Use an account with sufficient permissions (superuser) to update your replication settings. 1. Open the `postgresql.conf` file. 2. Find the `wal_level` parameter. 3. Update the parameter value to `wal_level = logical`. If you already use replication slots, you may need to increase the limit on replication slots (`max_replication_slots`). The `max_wal_senders` parameter value must also be greater than or equal to `max_replication_slots`. 4. Restart the PostgreSQL server. For this input to make a successful connection to your database, also make sure that it allows replication connections. 1. Open the `pg_hba.conf` file. 2. Update this line. ```yaml host replication /32 md5 ``` Replace the following placeholders with your own values: - ``: The username from an account with superuser privileges. - ``: The IP address of the server where you are running Redpanda Connect. 3. Restart the PostgreSQL server. ## [](#choose-a-replication-mode)Choose a replication mode When you run a pipeline that uses the `postgres_cdc` input, Redpanda Connect connects to your PostgreSQL database and creates a replication slot. The replication slot uses a copy of the Write-Ahead Log (WAL) file to subscribe to changes in your database records as they are applied to the database. There are two replication modes you can choose from: snapshot mode and streaming mode. In snapshot mode, Redpanda Connect first takes a snapshot of the database and streams the contents before processing changes from the WAL. In streaming mode, Redpanda Connect directly processes changes from the WAL starting from the most recent changes without taking a snapshot first. For local testing, you can use the [example pipeline on this page](#example-pipeline), which runs in snapshot mode. ### [](#snapshot-mode)Snapshot mode If you set the [`stream_snapshot` field](#stream_snapshot) to `true`, Redpanda Connect: 1. Creates a snapshot of your database. 2. Streams the contents of the tables specified in the `postgres_cdc` input. 3. Starts processing changes in the WAL that occurred since the snapshot was taken, and streams them to Redpanda Connect. Once the initial replication process is complete, the snapshot is removed and the input keeps a connection open to the database so that it can receive data updates. If the pipeline restarts during the replication process, Redpanda Connect resumes processing data changes from where it left off. If there are other interruptions while the snapshot is taken, you may need to restart the snapshot process. For more information, see [Troubleshoot replication failures](#troubleshoot_replication_failures). ### [](#streaming-mode)Streaming mode If you set the [`stream_snapshot` field](#stream_snapshot) to `false`, Redpanda Connect starts processing data changes from the end of the WAL. If the pipeline restarts, Redpanda Connect resumes processing data changes from the last acknowledged position in the WAL. ## [](#monitor-the-replication-process)Monitor the replication process You can monitor the initial replication of data using the following metrics: | Metric name | Description | | --- | --- | | replication_lag_bytes | Indicates how far the connector is lagging behind the source database when processing the transaction log. | | postgres_snapshot_progress | Shows the progress of snapshot processing for each table. | ## [](#troubleshoot-replication-failures)Troubleshoot replication failures If the database snapshot fails, the replication slot has only an incomplete record of the existing data in your database. To maintain data integrity, you must drop the replication slot manually in your source database and run the Redpanda Connect pipeline again. ```SQL SELECT pg_drop_replication_slot(SLOT_NAME); ``` ## [](#receive-toast-and-deleted-values)Receive TOAST and deleted values For full visibility of all data updates, you can also choose to stream [TOAST](https://www.postgresql.org/docs/current/storage-toast.html) and deleted values. To enable this option, run the following query on your source database: ```SQL ALTER TABLE large_data REPLICA IDENTITY FULL; ``` ## [](#data-mappings)Data mappings The following table shows how selected PostgreSQL data types are mapped to data types supported in Redpanda Connect. All other data types are mapped to string values. | PostgreSQL data type | Bloblang value | | --- | --- | | TEXT, TIMESTAMP, UUID, VARCHAR | JSON strings, for example: this data | | BOOL | Boolean JSON fields, for example: true or false | | Numeric types (INT4) | JSON number types, for example: 1. | | JSONB | JSON objects, for example: { "message": "message text" } | | INTEGER[] | An array of integer values, for example: [1,2,3] | | TEXT[] | An array of string values, for example: ["value1", "value2", "value3"] | | INET | A string that contains an IP address, for example: "192.168.1.1" | | POINT | A string that represents a point in a two-dimensional plane, for example: (x, y) | | TSRANGE | A string that includes range bounds, for example: [2010-01-01 14:30, 2010-01-01 15:30) | | TSVECTOR | A string that includes vector data, for example: "'the':2 'question':3 'is':4" | ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `table`: The name of the database table from which the message originated. - `operation`: The type of database operation that generated the message, such as `read`, `insert`, `update`, `delete`, `begin` and `commit`. A `read` operation occurs when a snapshot of the database is processed. The `begin` and `commit` operations are only included if the `include_transaction_markers` field is set to `true`. - `lsn`: The [Log Sequence Number](https://www.postgresql.org/docs/current/datatype-pg-lsn.html) of each data update from the source PostgreSQL database. The `lsn` values are strings that can be sorted to determine the order in which data updates were written to the WAL. ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay rejected messages (negative acknowledgements) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#aws)`aws` AWS IAM authentication configuration for PostgreSQL instances. When enabled, IAM credentials are used to generate temporary authentication tokens instead of a static password. This is useful for connecting to Amazon RDS or Aurora PostgreSQL instances with IAM database authentication enabled. The generated tokens are valid for 15 minutes and are automatically refreshed. For more information about AWS credentials configuration, see the [credentials for AWS](../../../guides/cloud/aws/) guide. **Type**: `object` ### [](#aws-enabled)`aws.enabled` Enable AWS IAM authentication for PostgreSQL. When enabled, an IAM authentication token is generated and used as the password. **Type**: `bool` **Default**: `false` ### [](#aws-endpoint)`aws.endpoint` The PostgreSQL endpoint hostname (e.g., mydb.abc123.us-east-1.rds.amazonaws.com). **Type**: `string` ### [](#aws-id)`aws.id` The ID of credentials to use. **Type**: `string` ### [](#aws-region)`aws.region` The AWS region where the PostgreSQL instance is located. If no region is specified then the environment default will be used. **Type**: `string` ### [](#aws-role)`aws.role` Optional AWS IAM role ARN to assume for authentication. Alternatively, use `roles` array for role chaining instead. **Type**: `string` ### [](#aws-role_external_id)`aws.role_external_id` Optional external ID for the role assumption. Only used with the `role` field. Alternatively, use `roles` array for role chaining instead. **Type**: `string` ### [](#aws-roles)`aws.roles[]` Optional array of AWS IAM roles to assume for authentication. Roles can be assumed in sequence, enabling chaining for purposes such as cross-account access. Each role can optionally specify an external ID. **Type**: `object` ### [](#aws-roles-role)`aws.roles[].role` AWS IAM role ARN to assume. **Type**: `string` **Default**: `""` ### [](#aws-roles-role_external_id)`aws.roles[].role_external_id` Optional external ID for the role assumption. **Type**: `string` **Default**: `""` ### [](#aws-secret)`aws.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#aws-token)`aws.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#checkpoint_limit)`checkpoint_limit` The maximum number of messages that this input can process at a given time. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given log sequence number (LSN) is not acknowledged until all messages under that offset are delivered. **Type**: `int` **Default**: `1024` ### [](#dsn)`dsn` The data source name (DSN) of the PostgreSQL database from which you want to stream updates. Use the format `postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&…​]`. For example, if you wanted to disable SSL in a secure environment, you would add `sslmode=disable` to the connection string. **Type**: `string` ```yaml # Examples: dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable ``` ### [](#heartbeat_interval)`heartbeat_interval` The interval between heartbeat messages, which Redpanda Connect writes to the WAL using the `pg_logical_emit_message` function. Heartbeat messages are useful when you subscribe to data changes from tables with low activity, while other tables in the database have higher-frequency updates. Heartbeat messages allow Redpanda Connect to periodically acknowledge new messages even when no data updates occur. Each acknowledgement advances the committed point in the WAL, which ensures that PostgreSQL can safely reclaim older log segments, preventing excessive disk space usage. Set `heartbeat_interval` to `0s` to disable heartbeats. **Type**: `string` **Default**: `1h` ```yaml # Examples: heartbeat_interval: 0s # --- heartbeat_interval: 24h ``` ### [](#include_transaction_markers)`include_transaction_markers` When set to `true`, creates empty messages for `BEGIN` and `COMMIT` operations which start and complete each transaction. Messages with the `operation` metadata field set to `BEGIN` or `COMMIT` have null message payloads. **Type**: `bool` **Default**: `false` ### [](#max_parallel_snapshot_tables)`max_parallel_snapshot_tables` Specify the maximum number of tables that are processed in parallel when the initial snapshot of the source database is taken. **Type**: `int` **Default**: `1` ### [](#pg_standby_timeout)`pg_standby_timeout` Specify the standby timeout after which an idle connection is refreshed to keep the connection alive. **Type**: `string` **Default**: `10s` ```yaml # Examples: pg_standby_timeout: 30s ``` ### [](#pg_wal_monitor_interval)`pg_wal_monitor_interval` How often to report changes to the replication lag and write them to Redpanda Connect metrics. **Type**: `string` **Default**: `3s` ```yaml # Examples: pg_wal_monitor_interval: 6s ``` ### [](#schema)`schema` The PostgreSQL schema from which to replicate data. **Type**: `string` ```yaml # Examples: schema: public # --- schema: "MyCaseSensitiveSchemaNeedingQuotes" ``` ### [](#slot_name)`slot_name` The name of the PostgreSQL logical replication slot to use. If not provided, a random name is generated unless you create a replication slot manually before starting replication. > 📝 **NOTE** > > Starting from version 4.48.1, Redpanda Connect no longer adds the prefix `rs_` to the names of replication slots it creates. To continue using an existing replication slot after upgrading, manually add the `rs_` prefix to the slot name. **Type**: `string` ```yaml # Examples: slot_name: my_test_slot ``` ### [](#snapshot_batch_size)`snapshot_batch_size` The number of table rows to fetch in each batch when querying the snapshot. This option is only available when `stream_snapshot` is set to `true`. **Type**: `int` **Default**: `1000` ```yaml # Examples: snapshot_batch_size: 10000 ``` ### [](#stream_snapshot)`stream_snapshot` When set to `true`, this input streams a snapshot of all existing data in the source database before streaming data changes. To use this setting, all database tables that you want to replicate _must_ have a primary key. **Type**: `bool` **Default**: `false` ```yaml # Examples: stream_snapshot: true ``` ### [](#tables)`tables[]` A list of database table names to include in the snapshot and logical replication. Specify each table name as a separate item. **Type**: `array` ```yaml # Examples: tables: - my_table_1 - "MyCaseSensitiveTableNeedingQuotes" ``` ### [](#temporary_slot)`temporary_slot` If set to `true`, the input creates a temporary replication slot that is automatically dropped when the connection to your source database is closed. You might use this option to: - Avoid data accumulating in the replication slot when a pipeline is paused or stopped - Test the connector If the pipeline is restarted, another data snapshot is taken before data updates are streamed. **Type**: `bool` **Default**: `false` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#unchanged_toast_value)`unchanged_toast_value` Specify the value to emit when unchanged [TOAST values](#receive-toast-and-deleted-values) appear in the message stream. Unchanged values occur for data updates and deletes when `REPLICA IDENTITY` is not set to `FULL`. **Type**: `unknown` **Default**: ```yaml null ``` ```yaml # Examples: unchanged_toast_value: __redpanda_connect_unchanged_toast_value__ ``` ## [](#example-pipeline)Example pipeline You can run the following pipeline locally to check that data updates are streamed from your source database to Redpanda Connect. All transactions are written to stdout. ```yml input: label: "postgres_cdc" postgres_cdc: dsn: postgres://user:password@host:port/dbname include_transaction_markers: false slot_name: test_slot_native_decoder snapshot_batch_size: 100000 stream_snapshot: true temporary_slot: true schema: schema_name tables: - table_name cache_resources: - label: data_caching file: directory: /tmp/cache output: label: main stdout: {} ``` --- # Page 84: pulsar **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/pulsar.md --- # pulsar --- title: pulsar latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/pulsar page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/pulsar.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/pulsar.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/pulsar/)[Output](/redpanda-connect/components/outputs/pulsar/) **Available in:** Self-Managed Reads messages from an Apache Pulsar server. Introduced in version 3.43.0. #### Common ```yml inputs: label: "" pulsar: url: "" # No default (required) topics: [] # No default (optional) topics_pattern: "" # No default (optional) subscription_name: "" # No default (required) subscription_type: shared subscription_initial_position: latest tls: root_cas_file: "" ``` #### Advanced ```yml inputs: label: "" pulsar: url: "" # No default (required) topics: [] # No default (optional) topics_pattern: "" # No default (optional) subscription_name: "" # No default (required) subscription_type: shared subscription_initial_position: latest tls: root_cas_file: "" auth: oauth2: enabled: false audience: "" issuer_url: "" scope: "" private_key_file: "" token: enabled: false token: "" ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: ```text - pulsar_message_id - pulsar_key - pulsar_ordering_key - pulsar_event_time_unix - pulsar_publish_time_unix - pulsar_topic - pulsar_producer_name - pulsar_redelivery_count - All properties of the message ``` You can access these metadata fields using [function interpolation](../../../configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of Pulsar authentication methods. Requires version 3.60.0 or later. **Type**: `object` ### [](#auth-oauth2)`auth.oauth2` Parameters for Pulsar OAuth2 authentication. **Type**: `object` ### [](#auth-oauth2-audience)`auth.oauth2.audience` OAuth2 audience. **Type**: `string` **Default**: `""` ### [](#auth-oauth2-enabled)`auth.oauth2.enabled` Whether OAuth2 is enabled. **Type**: `bool` **Default**: `false` ### [](#auth-oauth2-issuer_url)`auth.oauth2.issuer_url` OAuth2 issuer URL. **Type**: `string` **Default**: `""` ### [](#auth-oauth2-private_key_file)`auth.oauth2.private_key_file` The path to a file containing a private key. **Type**: `string` **Default**: `""` ### [](#auth-oauth2-scope)`auth.oauth2.scope` OAuth2 scope to request. **Type**: `string` **Default**: `""` ### [](#auth-token)`auth.token` Parameters for Pulsar Token authentication. **Type**: `object` ### [](#auth-token-enabled)`auth.token.enabled` Whether Token Auth is enabled. **Type**: `bool` **Default**: `false` ### [](#auth-token-token)`auth.token.token` Actual base64 encoded token. **Type**: `string` **Default**: `""` ### [](#subscription_initial_position)`subscription_initial_position` Specify the subscription initial position for this consumer. **Type**: `string` **Default**: `latest` **Options**: `latest`, `earliest` ### [](#subscription_name)`subscription_name` Specify the subscription name for this consumer. **Type**: `string` ### [](#subscription_type)`subscription_type` Specify the subscription type for this consumer. > > 📝 **NOTE** > > > > Using a `key_shared` subscription type will _allow out-of-order delivery_ since nack-ing messages sets non-zero nack delivery delay - this can potentially cause consumers to stall. See [Pulsar documentation](https://pulsar.apache.org/docs/en/2.8.1/concepts-messaging/#negative-acknowledgement) and [this Github issue](https://github.com/apache/pulsar/issues/12208) for more details. **Type**: `string` **Default**: `shared` **Options**: `shared`, `key_shared`, `failover`, `exclusive` ### [](#tls)`tls` Specify the path to a custom CA certificate to trust broker TLS service. **Type**: `object` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#topics)`topics[]` A list of topics to subscribe to. This or topics\_pattern must be set. **Type**: `array` ### [](#topics_pattern)`topics_pattern` A regular expression matching the topics to subscribe to. This or topics must be set. **Type**: `string` ### [](#url)`url` A URL to connect to. **Type**: `string` ```yaml # Examples: url: pulsar://localhost:6650 # --- url: pulsar://pulsar.us-west.example.com:6650 # --- url: pulsar+ssl://pulsar.us-west.example.com:6651 ``` --- # Page 85: read_until **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/read_until.md --- # read\_until --- title: read_until latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/read_until page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/read_until.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/read_until.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/read_until/ "View the Cloud version of this component"), Self-Managed Reads messages from a child input until a consumed message passes a [Bloblang query](../../../guides/bloblang/about/), at which point the input closes. It is also possible to configure a timeout after which the input is closed if no new messages arrive in that period. ```yml inputs: label: "" read_until: input: "" # No default (required) check: "" # No default (optional) idle_timeout: "" # No default (optional) restart_input: false ``` Messages are read continuously while the query check returns false, when the query returns true the message that triggered the check is sent out and the input is closed. Use this to define inputs where the stream should end once a certain message appears. If the idle timeout is configured, the input will be closed if no new messages arrive after that period of time. Use this field if you want to empty out and close an input that doesn’t have a logical end. Sometimes inputs close themselves. For example, when the `file` input type reaches the end of a file it will shut down. By default this type will also shut down. If you wish for the input type to be restarted every time it shuts down until the query check is met then set `restart_input` to `true`. ## [](#metadata)Metadata A metadata key `benthos_read_until` containing the value `final` is added to the first part of the message that triggers the input to stop. ## [](#fields)Fields ### [](#check)`check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether the input should now be closed. **Type**: `string` ```yaml # Examples: check: this.type == "foo" # --- check: count("messages") >= 100 ``` ### [](#idle_timeout)`idle_timeout` The maximum amount of time without receiving new messages after which the input is closed. **Type**: `string` ```yaml # Examples: idle_timeout: 5s ``` ### [](#input)`input` The child input to consume from. **Type**: `input` ### [](#restart_input)`restart_input` Whether the input should be reopened if it closes itself before the condition has resolved to true. **Type**: `bool` **Default**: `false` ## [](#examples)Examples ### [](#consume-n-messages)Consume N Messages A common reason to use this input is to consume only N messages from an input and then stop. This can easily be done with the [`count` function](../../../guides/bloblang/functions/#count): ```yaml # Only read 100 messages, and then exit. input: read_until: check: count("messages") >= 100 input: kafka: addresses: [ TODO ] topics: [ foo, bar ] consumer_group: foogroup ``` ### [](#read-from-a-kafka-and-close-when-empty)Read from a kafka and close when empty A common reason to use this input is a job that consumes all messages and exits once its empty: ```yaml # Consumes all messages and exit when the last message was consumed 5s ago. input: read_until: idle_timeout: 5s input: kafka: addresses: [ TODO ] topics: [ foo, bar ] consumer_group: foogroup ``` --- # Page 86: redis_list **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/redis_list.md --- # redis\_list --- title: redis_list latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/redis_list page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/redis_list.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/redis_list.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/redis_list/)[Output](/redpanda-connect/components/outputs/redis_list/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/redis_list/ "View the Cloud version of this component"), Self-Managed Pops messages from the beginning of a Redis list using the BLPop command. #### Common ```yml inputs: label: "" redis_list: url: "" # No default (required) key: "" # No default (required) auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" redis_list: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] key: "" # No default (required) auto_replay_nacks: true max_in_flight: 0 timeout: 5s command: blpop ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#client_name)`client_name` Set the client name for the Redis connection. Requires version 4.82.0 or later. **Type**: `string` **Default**: `redpanda-connect` ### [](#command)`command` The command used to pop elements from the Redis list Requires version 4.22.0 or later. **Type**: `string` **Default**: `blpop` **Options**: `blpop`, `brpop` ### [](#key)`key` The key of a list to read from. **Type**: `string` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#max_in_flight)`max_in_flight` Optionally sets a limit on the number of messages that can be flowing through a Redpanda Connect stream pending acknowledgment from the input at any given time. Once a message has been either acknowledged or rejected (nacked) it is no longer considered pending. If the input produces logical batches then each batch is considered a single count against the maximum. **WARNING**: Batching policies at the output level will stall if this field limits the number of messages below the batching threshold. Zero (default) or lower implies no limit. Requires version 4.9.0 or later. **Type**: `int` **Default**: `0` ### [](#timeout)`timeout` The length of time to poll for new messages before reattempting. **Type**: `string` **Default**: `5s` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 87: redis_pubsub **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/redis_pubsub.md --- # redis\_pubsub --- title: redis_pubsub latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/redis_pubsub page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/redis_pubsub.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/redis_pubsub.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/redis_pubsub/)[Output](/redpanda-connect/components/outputs/redis_pubsub/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/redis_pubsub/ "View the Cloud version of this component"), Self-Managed Consume from a Redis publish/subscribe channel using either the SUBSCRIBE or PSUBSCRIBE commands. #### Common ```yml inputs: label: "" redis_pubsub: url: "" # No default (required) channels: [] # No default (required) use_patterns: false auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" redis_pubsub: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] channels: [] # No default (required) use_patterns: false auto_replay_nacks: true ``` In order to subscribe to channels using the `PSUBSCRIBE` command set the field `use_patterns` to `true`, then you can include glob-style patterns in your channel names. For example: - `h?llo` subscribes to hello, hallo and hxllo - `h*llo` subscribes to hllo and heeeello - `h[ae]llo` subscribes to hello and hallo, but not hillo Use `\` to escape special characters if you want to match them verbatim. ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#channels)`channels[]` A list of channels to consume from. **Type**: `array` ### [](#client_name)`client_name` Set the client name for the Redis connection. Requires version 4.82.0 or later. **Type**: `string` **Default**: `redpanda-connect` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` ### [](#use_patterns)`use_patterns` Whether to use the PSUBSCRIBE command, allowing for glob-style patterns within target channel names. **Type**: `bool` **Default**: `false` --- # Page 88: redis_scan **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/redis_scan.md --- # redis\_scan --- title: redis_scan latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/redis_scan page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/redis_scan.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/redis_scan.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/redis_scan/ "View the Cloud version of this component"), Self-Managed Scans the set of keys in the current selected database and gets their values, using the Scan and Get commands. Introduced in version 4.27.0. #### Common ```yml inputs: label: "" redis_scan: url: "" # No default (required) auto_replay_nacks: true match: "" ``` #### Advanced ```yml inputs: label: "" redis_scan: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] auto_replay_nacks: true match: "" ``` Optionally, iterates only elements matching a blob-style pattern. For example: - `**foo**` iterates only keys which contain `foo` in it. - `foo*` iterates only keys starting with `foo`. This input generates a message for each key value pair in the following format: ```json {"key":"foo","value":"bar"} ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#client_name)`client_name` Set the client name for the Redis connection. Requires version 4.82.0 or later. **Type**: `string` **Default**: `redpanda-connect` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#match)`match` Iterates only elements matching the optional glob-style pattern. By default, it matches all elements. **Type**: `string` **Default**: `""` ```yaml # Examples: match: * # --- match: 1* # --- match: foo* # --- match: foo # --- match: *4* ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 89: redis_streams **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/redis_streams.md --- # redis\_streams --- title: redis_streams latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/redis_streams page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/redis_streams.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/redis_streams.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/redis_streams/)[Output](/redpanda-connect/components/outputs/redis_streams/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/redis_streams/ "View the Cloud version of this component"), Self-Managed Pulls messages from Redis (v5.0+) streams with the XREADGROUP command. The `client_id` should be unique for each consumer of a group. #### Common ```yml inputs: label: "" redis_streams: url: "" # No default (required) body_key: body streams: [] # No default (required) auto_replay_nacks: true limit: 10 client_id: "" consumer_group: "" ``` #### Advanced ```yml inputs: label: "" redis_streams: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] body_key: body streams: [] # No default (required) auto_replay_nacks: true limit: 10 client_id: "" consumer_group: "" create_streams: true start_from_oldest: true commit_period: 1s timeout: 1s ``` Redis stream entries are key/value pairs, as such it is necessary to specify the key that contains the body of the message. All other keys/value pairs are saved as metadata fields. ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#body_key)`body_key` The field key to extract the raw message from. All other keys will be stored in the message as metadata. **Type**: `string` **Default**: `body` ### [](#client_id)`client_id` An identifier for the client connection. **Type**: `string` **Default**: `""` ### [](#client_name)`client_name` Set the client name for the Redis connection. Requires version 4.82.0 or later. **Type**: `string` **Default**: `redpanda-connect` ### [](#commit_period)`commit_period` The period of time between each commit of the current offset. Offsets are always committed during shutdown. **Type**: `string` **Default**: `1s` ### [](#consumer_group)`consumer_group` An identifier for the consumer group of the stream. **Type**: `string` **Default**: `""` ### [](#create_streams)`create_streams` Create subscribed streams if they do not exist (MKSTREAM option). **Type**: `bool` **Default**: `true` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#limit)`limit` The maximum number of messages to consume from a single request. **Type**: `int` **Default**: `10` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#start_from_oldest)`start_from_oldest` If an offset is not found for a stream, determines whether to consume from the oldest available offset, otherwise messages are consumed from the latest offset. **Type**: `bool` **Default**: `true` ### [](#streams)`streams[]` A list of streams to consume from. **Type**: `array` ### [](#timeout)`timeout` The length of time to poll for new messages before reattempting. **Type**: `string` **Default**: `1s` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 90: redpanda_common **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/redpanda_common.md --- # redpanda\_common --- title: redpanda_common latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/redpanda_common page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/redpanda_common.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/redpanda_common.adoc page-git-created-date: "2024-11-19" page-git-modified-date: "2025-11-03" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/redpanda_common/)[Output](/redpanda-connect/components/outputs/redpanda_common/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/redpanda_common/ "View the Cloud version of this component"), Self-Managed **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. > ⚠️ **WARNING: Deprecated in 4.68.0** > > Deprecated in 4.68.0 > > This component is deprecated and will be removed in the next major version release. Please consider moving onto the unified [`redpanda` input](../redpanda/) and [`redpanda` output](../../outputs/redpanda/) components. Consumes data from a Redpanda (Kafka) broker, using credentials from a common `redpanda` configuration block. To avoid duplicating Redpanda cluster credentials in your `redpanda_common` input, output, or any other components in your data pipeline, you can use a single [`redpanda` configuration block](../../redpanda/about/). For more details, see the [Pipeline example](#pipeline-example). Introduced in version 4.39.0. > 📝 **NOTE** > > If you need to move topic data between Redpanda clusters or other Apache Kafka clusters, consider using the [`redpanda` input](../redpanda/) and [output](../../outputs/redpanda/) instead. #### Common ```yml inputs: label: "" redpanda_common: topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" redpanda_common: topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) rack_id: "" instance_id: "" rebalance_timeout: 45s session_timeout: 1m heartbeat_interval: 3s start_offset: earliest fetch_max_bytes: 50MiB fetch_max_wait: 5s fetch_min_bytes: 1B fetch_max_partition_bytes: 1MiB transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) commit_period: 5s partition_buffer_bytes: 1MB topic_lag_refresh_period: 5s max_yield_batch_bytes: 32KB auto_replay_nacks: true timely_nacks_maximum_wait: "" # No default (optional) ``` ## [](#pipeline-example)Pipeline example This data pipeline reads data from `topic_A` and `topic_B` on a Redpanda cluster, and then writes the data to `topic_C` on the same cluster. The cluster details are configured within the `redpanda` configuration block, so you only need to configure them once. This is a useful feature when you have multiple inputs and outputs in the same data pipeline that need to connect to the same cluster. ```none input: redpanda_common: topics: [ topic_A, topic_B ] output: redpanda_common: topic: topic_C key: ${! @id } redpanda: seed_brokers: [ "127.0.0.1:9092" ] tls: enabled: true sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ## [](#consumer-groups)Consumer groups When you specify a consumer group in your configuration, this input consumes one or more topics and automatically balances the topic partitions across any other connected clients with the same consumer group. Otherwise, topics are consumed in their entirety or with explicit partitions. ### [](#delivery-guarantees)Delivery guarantees If you choose to use consumer groups, the offsets of records received by Redpanda Connect are committed automatically. In the event of restarts, this input uses the committed offsets to resume data consumption where it left off. Redpanda Connect guarantees at-least-once delivery. Records are only confirmed as delivered when all downstream outputs that a record is routed to have also confirmed delivery. ## [](#ordering)Ordering To preserve the order of topic partitions: - Records consumed from each partition are processed and delivered in the order that they are received - Only one batch of records of a given partition is processed at a time This approach means that although records from different partitions may be processed in parallel, records from the same partition are processed in sequential order. ### [](#delivery-errors)Delivery errors The order in which records are delivered may be disrupted by delivery errors and any error-handling mechanisms that start up. Redpanda Connect uses at-least-once delivery unless instructed otherwise, and this includes reattempting delivery of data when the ordering of that data is no longer guaranteed. For example, a batch of records is sent to an output broker and only a subset of records are delivered. In this scenario, Redpanda Connect (by default) attempts to deliver the records that failed, even though these delivery failures may have been sent before records that were delivered successfully. #### [](#use-a-fallback-output)Use a fallback output To prevent delivery errors from disrupting the order of records, you must specify a [`fallback`](../../outputs/fallback/) output in your pipeline configuration. When adding a `fallback` output, it is good practice to set the `auto_retry_nacks` field to `false`. This also improves the throughput of your pipeline. For example, the following configuration includes a `fallback` output. If Redpanda Connect fails to write delivery errors to the `foo` topic, it then attempts to write them into a dead letter queue topic (`foo_dlq`), which is retried indefinitely as a way to apply back pressure. ```yaml output: fallback: - redpanda_common: topic: foo - retry: output: redpanda_common: topic: foo_dlq ``` ## [](#batching)Batching Records are processed and delivered from each partition in the same batches as they are received from brokers. Batch sizes are dynamically sized in order to optimize throughput, but you can tune them further using the following configuration fields: - `fetch_max_partition_bytes` - `fetch_max_bytes` You can break batches down further using the [`split`](../../processors/split/) processor. ## [](#metrics)Metrics This input emits a `redpanda_lag` metric with `topic` and `partition` labels for each consumed topic. The metric records the number of produced messages that remain to be read from each topic/partition pair by the specified consumer group. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `kafka_key` - `kafka_topic` - `kafka_partition` - `kafka_offset` - `kafka_lag` - `kafka_timestamp_ms` - `kafka_timestamp_unix` - `kafka_tombstone_message` - All record headers ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay messages that are rejected (nacked) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#commit_period)`commit_period` The period of time between each commit of the current partition offsets. Offsets are always committed during shutdown. **Type**: `string` **Default**: `5s` ### [](#consumer_group)`consumer_group` An optional consumer group. When this value is specified: - The partitions of any topics, specified in the `topics` field, are automatically distributed across consumers sharing a consumer group - Partition offsets are automatically committed and resumed under this name Consumer groups are not supported when you specify explicit partitions to consume from in the `topics` field. **Type**: `string` ### [](#fetch_max_bytes)`fetch_max_bytes` The maximum number of bytes that a broker tries to send during a fetch. If individual records are larger than the `fetch_max_bytes` value, brokers will still send them. **Type**: `string` **Default**: `50MiB` ### [](#fetch_max_partition_bytes)`fetch_max_partition_bytes` The maximum number of bytes that are consumed from a single partition in a fetch request. This field is equivalent to the Java setting `fetch.max.partition.bytes`. If a single batch is larger than the `fetch_max_partition_bytes` value, the batch is still sent so that the client can make progress. **Type**: `string` **Default**: `1MiB` ### [](#fetch_max_wait)`fetch_max_wait` The maximum period of time a broker can wait for a fetch response to reach the required minimum number of bytes (`fetch_min_bytes`). **Type**: `string` **Default**: `5s` ### [](#fetch_min_bytes)`fetch_min_bytes` The minimum number of bytes that a broker tries to send during a fetch. This field is equivalent to the Java setting `fetch.min.bytes`. **Type**: `string` **Default**: `1B` ### [](#heartbeat_interval)`heartbeat_interval` When you specify a `consumer_group`, `heartbeat_interval` sets how frequently a consumer group member should send heartbeats to Apache Kafka. Apache Kafka uses heartbeats to make sure that a group member’s session is active. You must set `heartbeat_interval` to less than one-third of `session_timeout`. This field is equivalent to the Java `heartbeat.interval.ms` setting and accepts Go duration format strings such as `10s` or `2m`. **Type**: `string` **Default**: `3s` ### [](#instance_id)`instance_id` When you specify a [`consumer_group`](#consumer_group), assign a unique value to `instance_id` to define the group’s static membership, which can prevent unnecessary rebalances during reconnections. When you assign an instance ID, the client does not automatically leave the consumer group when it disconnects. To remove the client, you must use an external admin command on behalf of the instance ID. **Type**: `string` **Default**: `""` ### [](#max_yield_batch_bytes)`max_yield_batch_bytes` The maximum size (in bytes) for each batch yielded by this input. This value must be less than or equal to the `partition_buffer_bytes`. If using Redpanda output, this value should not be greater than the `max_message_bytes` option value (1MB by default), and for high-throughput scenarios they should be equal. **Type**: `string` **Default**: `32KB` ### [](#partition_buffer_bytes)`partition_buffer_bytes` A buffer size (in bytes) for each consumed partition, which allows the internal queuing of records before they are flushed. Increasing this value may improve throughput but results in higher memory utilization. Each buffer can grow slightly beyond this value. **Type**: `string` **Default**: `1MB` ### [](#rack_id)`rack_id` A rack specifies where the client is physically located, and changes fetch requests to consume from the closest replica as opposed to the leader replica. **Type**: `string` **Default**: `""` ### [](#rebalance_timeout)`rebalance_timeout` When you specify a [`consumer_group`](#consumer_group), `rebalance_timeout` sets a time limit for all consumer group members to complete their work and commit offsets after a rebalance has begun. The timeout excludes the time taken to detect a failed or late heartbeat, which indicates a rebalance is required. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `45s` ### [](#regexp_topics_exclude)`regexp_topics_exclude[]` A list of regular expression patterns for excluding topics when regex mode is enabled (using `regexp_topics_include` or the deprecated `regexp_topics` boolean). Topics matching any of these patterns will be excluded from consumption, even if they match include patterns. Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so use `^` and `$` for exact matching. Exclude patterns are applied after include patterns, providing fine-grained control over topic selection. Example: `regexp_topics_exclude: ["^_", ".**-temp$", ".**-test.*"]` excludes topics starting with underscore, ending with `-temp`, or containing `-test`. **Type**: `array` ### [](#regexp_topics_include)`regexp_topics_include[]` A list of regular expression patterns for matching topics to consume from. When specified, the client will periodically refresh the list of matching topics based on the `metadata_max_age` interval. Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so `logs_.` **matches `my-logs_events` and `logs_errors`. Use `^logs_.`**`$` to match only topics starting with `logs_`. This field enables regex mode (replacing the deprecated `regexp_topics` boolean) and cannot be used together with explicit `topics` lists. Use `regexp_topics_exclude` to filter out specific patterns from the matched topics. Example: `regexp_topics_include: ["events_.**", "logs_.**"]` consumes from all topics starting with `events_` or `logs_`. **Type**: `array` ```yaml # Examples: regexp_topics_include: - logs_.* - metrics_.* # --- regexp_topics_include: - "events_[0-9]+" ``` ### [](#session_timeout)`session_timeout` When you specify a `consumer_group`, `session_timeout` sets the maximum interval between heartbeats sent by a consumer group member to the broker. If a broker doesn’t receive a heartbeat from a group member before the timeout expires, it removes the member from the consumer group and initiates a rebalance. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `1m` ### [](#start_offset)`start_offset` Specify the offset from which this input starts or restarts consuming messages. Restarts occur when the `OffsetOutOfRange` error is seen during a fetch. **Type**: `string` **Default**: `earliest` | Option | Summary | | --- | --- | | committed | Prevents consuming a partition in a group if the partition has no prior commits. Corresponds to Kafka’s auto.offset.reset=none option | | earliest | Start from the earliest offset. Corresponds to Kafka’s auto.offset.reset=earliest option. | | latest | Start from the latest offset. Corresponds to Kafka’s auto.offset.reset=latest option. | ### [](#timely_nacks_maximum_wait)`timely_nacks_maximum_wait` EXPERIMENTAL: Specify a maximum period of time in which each message can be consumed and awaiting either acknowledgement or rejection before rejection is instead forced. This can be useful for avoiding situations where certain downstream components can result in blocked confirmation of delivery that exceeds SLAs. Accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` ### [](#topic_lag_refresh_period)`topic_lag_refresh_period` The interval between refresh cycles. During each cycle, this input queries the Redpanda Connect server to calculate the topic lag minus the number of produced messages that remain to be read from each topic/partition pair by the specified consumer group. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `5s` ### [](#topics)`topics[]` A list of topics to consume from. Use commas to separate multiple topics in a single element. When a `consumer_group` is specified, partitions are automatically distributed across consumers of a topic. Otherwise, all partitions are consumed. Alternatively, you can specify explicit partitions to consume by using a colon after the topic name. For example, `foo:0` would consume the partition `0` of the topic foo. This syntax supports ranges. For example, `foo:0-10` would consume partitions `0` through to `10` inclusive. It is also possible to specify an explicit offset to consume from by adding another colon after the partition. For example, `foo:0:10` would consume the partition `0` of the topic `foo` starting from the offset `10`. If the offset is not present (or remains unspecified) then the field `start_offset` determines which offset to start from. **Type**: `array` ```yaml # Examples: topics: - foo - bar # --- topics: - things.* # --- topics: - "foo,bar" # --- topics: - "foo:0" - "bar:1" - "bar:3" # --- topics: - "foo:0,bar:1,bar:3" # --- topics: - "foo:0-5" ``` ### [](#transaction_isolation_level)`transaction_isolation_level` The isolation level for handling transactional messages. This setting determines how transactions are processed and affects data consistency guarantees. **Type**: `string` **Default**: `read_uncommitted` | Option | Summary | | --- | --- | | read_committed | If set, only committed transactional records are processed. | | read_uncommitted | If set, then uncommitted records are processed. | --- # Page 91: redpanda_migrator **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/redpanda_migrator.md --- # redpanda\_migrator --- title: redpanda_migrator latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/redpanda_migrator page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/redpanda_migrator.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/redpanda_migrator.adoc page-git-created-date: "2024-10-02" page-git-modified-date: "2025-12-03" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/redpanda_migrator/)[Output](/redpanda-connect/components/outputs/redpanda_migrator/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/redpanda_migrator/ "View the Cloud version of this component"), Self-Managed Unified Kafka consumer for migrating data between Kafka/Redpanda clusters. Use this input with the [`redpanda_migrator` output](../../outputs/redpanda_migrator/) to safely transfer topic data, ACLs, schemas, and consumer group offsets between clusters. This component is designed for migration scenarios. Introduced in version 4.67.5. #### Common ```yml inputs: label: "" redpanda_migrator: seed_brokers: [] # No default (required) topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) schema_registry: url: "" # No default (required) timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" redpanda_migrator: seed_brokers: [] # No default (required) client_id: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) metadata_max_age: 1m request_timeout_overhead: 10s conn_idle_timeout: 20s tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) rack_id: "" instance_id: "" rebalance_timeout: 45s session_timeout: 1m heartbeat_interval: 3s start_offset: earliest fetch_max_bytes: 50MiB fetch_max_wait: 5s fetch_min_bytes: 1B fetch_max_partition_bytes: 1MiB transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) commit_period: 5s partition_buffer_bytes: 1MB topic_lag_refresh_period: 5s max_yield_batch_bytes: 32KB schema_registry: url: "" # No default (required) timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} auto_replay_nacks: true ``` The `redpanda_migrator` input: - Reads a batch of messages from a broker. - Waits for the `redpanda_migrator` output to acknowledge the writes before updating the Kafka consumer group offset. - Provides the same delivery guarantees and ordering semantics as the [`redpanda` input](../redpanda/). Specify a consumer group to make this input consume one or more topics and automatically balance the topic partitions across any other connected clients with the same consumer group. Otherwise, topics are consumed in their entirety or with explicit partitions. This input requires a corresponding `redpanda_migrator` output in the same pipeline. Each pipeline must have both input and output components configured. For capabilities, guarantees, scheduling, and examples, see the output documentation. ## [](#requirements)Requirements - Must be paired with a `redpanda_migrator` output in the same pipeline. - Requires access to a source Kafka or Redpanda cluster. - Consumer group configuration is recommended for partition balancing. ## [](#multiple-migrator-pairs)Multiple migrator pairs When using multiple migrator pairs in a single pipeline, coordination is based on the `label` field. The label of the input and output must match exactly for correct pairing. If labels do not match, migration fails for that pair. ## [](#performance-tuning-for-high-throughput)Performance tuning for high throughput For workloads with high message rates or large messages, adjust the following settings to optimize throughput: On this input component: - `partition_buffer_bytes`: Set to 2MB to increase per-partition buffer size - `max_yield_batch_bytes`: Set to 1MB to allow larger batches to be yielded On the paired `redpanda_migrator` output component: - `max_in_flight`: Set to the total number of partitions being copied in parallel (up to all partitions in the cluster) > 📝 **NOTE** > > Setting `max_yield_batch_bytes` over 1MB is counter-productive unless you change the broker settings to allow bigger messages or batches. The `partition_buffer_bytes` setting allows for partition readahead. ## [](#metrics)Metrics This input emits an `input_redpanda_migrator_lag` metric with `topic` and `partition` labels for each consumed topic. This metric records the number of produced messages that remain to be read from each topic/partition pair by the specified consumer group. Monitor this metric to track migration progress and detect bottlenecks. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - kafka\_key - kafka\_topic - kafka\_partition - kafka\_offset - kafka\_lag - kafka\_timestamp\_ms - kafka\_timestamp\_unix - All record headers ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay messages that are rejected (nacked) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#client_id)`client_id` An identifier for the client connection. **Type**: `string` **Default**: `redpanda-connect` ### [](#commit_period)`commit_period` The period of time between each commit of the current partition offsets. Offsets are always committed during shutdown. **Type**: `string` **Default**: `5s` ### [](#conn_idle_timeout)`conn_idle_timeout` The maximum duration that connections can remain idle before they are automatically closed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `20s` ### [](#consumer_group)`consumer_group` An optional consumer group. When specified, the partitions of specified topics are automatically distributed across consumers sharing a consumer group, and partition offsets are automatically committed and resumed under this name. Consumer groups are not supported when explicit partitions are specified to consume from in the `topics` field. **Type**: `string` ### [](#fetch_max_bytes)`fetch_max_bytes` The maximum number of bytes that a broker tries to send during a fetch. If individual records are larger than the `fetch_max_bytes` value, brokers still send them. **Type**: `string` **Default**: `50MiB` ### [](#fetch_max_partition_bytes)`fetch_max_partition_bytes` The maximum number of bytes that are consumed from a single partition in a fetch request. This field is equivalent to the Java setting `fetch.max.partition.bytes`. If a single batch is larger than the `fetch_max_partition_bytes` value, the batch is still sent so that the client can make progress. **Type**: `string` **Default**: `1MiB` ### [](#fetch_max_wait)`fetch_max_wait` The maximum period of time a broker can wait for a fetch response to reach the required minimum number of bytes (`fetch_min_bytes`). **Type**: `string` **Default**: `5s` ### [](#fetch_min_bytes)`fetch_min_bytes` The minimum number of bytes that a broker tries to send during a fetch. This field is equivalent to the Java setting `fetch.min.bytes`. **Type**: `string` **Default**: `1B` ### [](#heartbeat_interval)`heartbeat_interval` When you specify a `consumer_group`, `heartbeat_interval` sets how frequently a consumer group member should send heartbeats to Apache Kafka. Apache Kafka uses heartbeats to make sure that a group member’s session is active. You must set `heartbeat_interval` to less than one-third of `session_timeout`. This field is equivalent to the Java `heartbeat.interval.ms` setting and accepts Go duration format strings such as `10s` or `2m`. **Type**: `string` **Default**: `3s` ### [](#instance_id)`instance_id` When you specify a [`consumer_group`](#consumer_group), assign a unique value to `instance_id` to define the group’s static membership, which can prevent unnecessary rebalances during reconnections. When you assign an instance ID, the client does not automatically leave the consumer group when it disconnects. To remove the client, you must use an external admin command on behalf of the instance ID. **Type**: `string` **Default**: `""` ### [](#max_yield_batch_bytes)`max_yield_batch_bytes` The maximum size (in bytes) for each batch yielded by this input. This value must be less than or equal to the `partition_buffer_bytes`. If using Redpanda output, this value should not be greater than the `max_message_bytes` option value (1MB by default), and for high-throughput scenarios they should be equal. **Type**: `string` **Default**: `32KB` ### [](#metadata_max_age)`metadata_max_age` The maximum period of time after which metadata is refreshed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Lower values provide more responsive topic and partition discovery but may increase broker load. Higher values reduce broker queries but can delay detection of topology changes. **Type**: `string` **Default**: `1m` ### [](#partition_buffer_bytes)`partition_buffer_bytes` A buffer size (in bytes) for each consumed partition, which allows the internal queuing of records before they are flushed. Increasing this value may improve throughput but results in higher memory utilization. Each buffer can grow slightly beyond this value. **Type**: `string` **Default**: `1MB` ### [](#rack_id)`rack_id` A rack identifier for this client. **Type**: `string` **Default**: `""` ### [](#rebalance_timeout)`rebalance_timeout` When you specify a [`consumer_group`](#consumer_group), `rebalance_timeout` sets a time limit for all consumer group members to complete their work and commit offsets after a rebalance has begun. The timeout excludes the time taken to detect a failed or late heartbeat, which indicates a rebalance is required. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `45s` ### [](#regexp_topics_exclude)`regexp_topics_exclude[]` A list of regular expression patterns for excluding topics when regex mode is enabled (using `regexp_topics_include` or the deprecated `regexp_topics` boolean). Topics matching any of these patterns will be excluded from consumption, even if they match include patterns. Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so use `^` and `$` for exact matching. Exclude patterns are applied after include patterns, providing fine-grained control over topic selection. Example: `regexp_topics_exclude: ["^_", ".**-temp$", ".**-test.*"]` excludes topics starting with underscore, ending with `-temp`, or containing `-test`. **Type**: `array` ### [](#regexp_topics_include)`regexp_topics_include[]` A list of regular expression patterns for matching topics to consume from. When specified, the client will periodically refresh the list of matching topics based on the `metadata_max_age` interval. Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so `logs_.` **matches `my-logs_events` and `logs_errors`. Use `^logs_.`**`$` to match only topics starting with `logs_`. This field enables regex mode (replacing the deprecated `regexp_topics` boolean) and cannot be used together with explicit `topics` lists. Use `regexp_topics_exclude` to filter out specific patterns from the matched topics. Example: `regexp_topics_include: ["events_.**", "logs_.**"]` consumes from all topics starting with `events_` or `logs_`. **Type**: `array` ```yaml # Examples: regexp_topics_include: - logs_.* - metrics_.* # --- regexp_topics_include: - "events_[0-9]+" ``` ### [](#request_timeout_overhead)`request_timeout_overhead` Grants an additional buffer or overhead to requests that have timeout fields defined. This field is based on the behavior of Apache Kafka’s `request.timeout.ms` parameter. **Type**: `string` **Default**: `10s` ### [](#sasl)`sasl[]` Specify one or more methods of SASL authentication, which are tried in order. If the broker supports the first mechanism, all connections use that mechanism. If the first mechanism fails, the client picks the first supported mechanism. Connections fail if the broker does not support any client mechanisms. **Type**: `object` ```yaml # Examples: sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ### [](#sasl-aws)`sasl[].aws` Contains AWS specific fields for when the `mechanism` is set to `AWS_MSK_IAM`. **Type**: `object` ### [](#sasl-aws-credentials)`sasl[].aws.credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](../../../guides/cloud/aws/). **Type**: `object` ### [](#sasl-aws-credentials-from_ec2_role)`sasl[].aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#sasl-aws-credentials-id)`sasl[].aws.credentials.id` The ID of credentials to use. **Type**: `string` ### [](#sasl-aws-credentials-profile)`sasl[].aws.credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#sasl-aws-credentials-role)`sasl[].aws.credentials.role` A role ARN to assume. **Type**: `string` ### [](#sasl-aws-credentials-role_external_id)`sasl[].aws.credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#sasl-aws-credentials-secret)`sasl[].aws.credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#sasl-aws-credentials-token)`sasl[].aws.credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#sasl-aws-endpoint)`sasl[].aws.endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#sasl-aws-region)`sasl[].aws.region` The AWS region to target. **Type**: `string` ### [](#sasl-aws-tcp)`sasl[].aws.tcp` TCP socket configuration. **Type**: `object` ### [](#sasl-aws-tcp-connect_timeout)`sasl[].aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-aws-tcp-keep_alive)`sasl[].aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#sasl-aws-tcp-keep_alive-count)`sasl[].aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#sasl-aws-tcp-keep_alive-idle)`sasl[].aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-keep_alive-interval)`sasl[].aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-tcp_user_timeout)`sasl[].aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-extensions)`sasl[].extensions` Key/value pairs to add to OAUTHBEARER authentication requests. **Type**: `string` ### [](#sasl-mechanism)`sasl[].mechanism` The SASL mechanism to use. **Type**: `string` | Option | Summary | | --- | --- | | AWS_MSK_IAM | AWS IAM based authentication as specified by the 'aws-msk-iam-auth' java library. | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. | | REDPANDA_CLOUD_SERVICE_ACCOUNT | Redpanda Cloud Service Account authentication when running in Redpanda Cloud. | | SCRAM-SHA-256 | SCRAM based authentication as specified in RFC5802. | | SCRAM-SHA-512 | SCRAM based authentication as specified in RFC5802. | | none | Disable sasl authentication | ### [](#sasl-password)`sasl[].password` A password to provide for PLAIN or SCRAM-\* authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#sasl-token)`sasl[].token` The token to use for a single session’s OAUTHBEARER authentication. **Type**: `string` **Default**: `""` ### [](#sasl-username)`sasl[].username` A username to provide for PLAIN or SCRAM-\* authentication. **Type**: `string` **Default**: `""` ### [](#schema_registry)`schema_registry` Configuration for schema registry integration. Enables migration of schema subjects, versions, and compatibility settings between clusters. **Type**: `object` ### [](#schema_registry-basic_auth)`schema_registry.basic_auth` Allows you to specify basic authentication. **Type**: `object` ### [](#schema_registry-basic_auth-enabled)`schema_registry.basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-basic_auth-password)`schema_registry.basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-basic_auth-username)`schema_registry.basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt)`schema_registry.jwt` Beta Allows you to specify JWT authentication. **Type**: `object` ### [](#schema_registry-jwt-claims)`schema_registry.jwt.claims` A value used to identify the claims that issued the JWT. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-enabled)`schema_registry.jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-jwt-headers)`schema_registry.jwt.headers` Add optional key/value headers to the JWT. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-private_key_file)`schema_registry.jwt.private_key_file` A file with the PEM encoded via PKCS1 or PKCS8 as private key. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt-signing_method)`schema_registry.jwt.signing_method` A method used to sign the token such as RS256, RS384, RS512 or EdDSA. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth)`schema_registry.oauth` Allows you to specify open authentication via OAuth version 1. **Type**: `object` ### [](#schema_registry-oauth-access_token)`schema_registry.oauth.access_token` A value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-access_token_secret)`schema_registry.oauth.access_token_secret` A secret provided in order to establish ownership of a given access token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_key)`schema_registry.oauth.consumer_key` A value used to identify the client to the service provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_secret)`schema_registry.oauth.consumer_secret` A secret used to establish ownership of the consumer key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-enabled)`schema_registry.oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-timeout)`schema_registry.timeout` HTTP client timeout for schema registry requests. **Type**: `string` **Default**: `5s` ### [](#schema_registry-tls)`schema_registry.tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#schema_registry-tls-client_certs)`schema_registry.tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#schema_registry-tls-client_certs-cert)`schema_registry.tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-cert_file)`schema_registry.tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key)`schema_registry.tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key_file)`schema_registry.tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-password)`schema_registry.tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#schema_registry-tls-enable_renegotiation)`schema_registry.tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-enabled)`schema_registry.tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-root_cas)`schema_registry.tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#schema_registry-tls-root_cas_file)`schema_registry.tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#schema_registry-tls-skip_cert_verify)`schema_registry.tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#schema_registry-url)`schema_registry.url` The base URL of the schema registry service. Required for schema migration functionality. **Type**: `string` ```yaml # Examples: url: http://localhost:8081 # --- url: https://schema-registry.example.com:8081 ``` ### [](#seed_brokers)`seed_brokers[]` A list of broker addresses to connect to in order. Use commas to separate multiple addresses in a single list item. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" # --- seed_brokers: - "foo:9092" - "bar:9092" # --- seed_brokers: - "foo:9092,bar:9092" ``` ### [](#session_timeout)`session_timeout` When you specify a `consumer_group`, `session_timeout` sets the maximum interval between heartbeats sent by a consumer group member to the broker. If a broker doesn’t receive a heartbeat from a group member before the timeout expires, it removes the member from the consumer group and initiates a rebalance. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `1m` ### [](#start_offset)`start_offset` Specify the offset from which this input starts or restarts consuming messages. Restarts occur when the `OffsetOutOfRange` error is seen during a fetch. **Type**: `string` **Default**: `earliest` | Option | Summary | | --- | --- | | committed | Prevents consuming a partition in a group if the partition has no prior commits. Corresponds to Kafka’s auto.offset.reset=none option | | earliest | Start from the earliest offset. Corresponds to Kafka’s auto.offset.reset=earliest option. | | latest | Start from the latest offset. Corresponds to Kafka’s auto.offset.reset=latest option. | ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic_lag_refresh_period)`topic_lag_refresh_period` The interval between refresh cycles. During each cycle, this input queries the Redpanda Connect server to calculate the topic lag minus the number of produced messages that remain to be read from each topic/partition pair by the specified consumer group. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `5s` ### [](#topics)`topics[]` A list of topics to consume from. Use commas to separate multiple topics in a single element. When a `consumer_group` is specified, partitions are automatically distributed across consumers of a topic. Otherwise, all partitions are consumed. Alternatively, you can specify explicit partitions to consume by using a colon after the topic name. For example, `foo:0` would consume the partition `0` of the topic foo. This syntax supports ranges. For example, `foo:0-10` would consume partitions `0` through to `10` inclusive. It is also possible to specify an explicit offset to consume from by adding another colon after the partition. For example, `foo:0:10` would consume the partition `0` of the topic `foo` starting from the offset `10`. If the offset is not present (or remains unspecified) then the field `start_offset` determines which offset to start from. **Type**: `array` ```yaml # Examples: topics: - foo - bar # --- topics: - things.* # --- topics: - "foo,bar" # --- topics: - "foo:0" - "bar:1" - "bar:3" # --- topics: - "foo:0,bar:1,bar:3" # --- topics: - "foo:0-5" ``` ### [](#transaction_isolation_level)`transaction_isolation_level` The isolation level for handling transactional messages. This setting determines how transactions are processed and affects data consistency guarantees. **Type**: `string` **Default**: `read_uncommitted` | Option | Summary | | --- | --- | | read_committed | If set, only committed transactional records are processed. | | read_uncommitted | If set, then uncommitted records are processed. | ## [](#troubleshooting)Troubleshooting - Ensure the input and output `label` fields match exactly. - Both input and output must be present in the pipeline. - Verify consumer group configuration for partition balancing. - Monitor the lag metric for stalled migration. ## [](#suggested-reading)Suggested reading - [`redpanda_migrator` output](../../outputs/redpanda_migrator/) - [Migrating from legacy components](../../../guides/migration/migrate-unified-redpanda-migrator/) --- # Page 92: redpanda **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/redpanda.md --- # redpanda --- title: redpanda latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/redpanda page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/redpanda.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/redpanda.adoc page-git-created-date: "2024-11-19" page-git-modified-date: "2025-10-06" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/redpanda/)[Cache](/redpanda-connect/components/caches/redpanda/)[Output](/redpanda-connect/components/outputs/redpanda/)[Tracer](/redpanda-connect/components/tracers/redpanda/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/redpanda/ "View the Cloud version of this component"), Self-Managed Consumes topic data from one or more Kafka brokers. Introduced in version 4.39.0. > 📝 **NOTE** > > If you intend to use multiple `redpanda` components in your data pipeline, which have a common set of credentials, consider using the [`redpanda_common` input](../redpanda_common/) and [output](../../outputs/redpanda_common/) instead. #### Common ```yml inputs: label: "" redpanda: seed_brokers: [] # No default (optional) topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" redpanda: seed_brokers: [] # No default (optional) client_id: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) metadata_max_age: 1m request_timeout_overhead: 10s conn_idle_timeout: 20s tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s topics: [] # No default (optional) regexp_topics_include: [] # No default (optional) regexp_topics_exclude: [] # No default (optional) rack_id: "" instance_id: "" rebalance_timeout: 45s session_timeout: 1m heartbeat_interval: 3s start_offset: earliest fetch_max_bytes: 50MiB fetch_max_wait: 5s fetch_min_bytes: 1B fetch_max_partition_bytes: 1MiB transaction_isolation_level: read_uncommitted consumer_group: "" # No default (optional) commit_period: 5s partition_buffer_bytes: 1MB topic_lag_refresh_period: 5s max_yield_batch_bytes: 32KB unordered_processing: enabled: false checkpoint_limit: 1024 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) auto_replay_nacks: true timely_nacks_maximum_wait: "" # No default (optional) extract_tracing_map: "" # No default (optional) ``` ## [](#consumer-groups)Consumer groups When you specify a consumer group in your configuration, this input consumes one or more topics and automatically balances the topic partitions across any other connected clients with the same consumer group. Otherwise, topics are consumed in their entirety or with explicit partitions. ## [](#delivery-guarantees)Delivery guarantees If you choose to use consumer groups, the offsets of records received by Redpanda Connect are committed automatically. In the event of restarts, this input uses the committed offsets to resume data consumption where it left off. Redpanda Connect guarantees at-least-once delivery. Records are only confirmed as delivered when all downstream outputs that a record is routed to have also confirmed delivery. ## [](#ordering)Ordering To preserve the order of topic partitions: - Records consumed from each partition are processed and delivered in the order that they are received - Only one batch of records of a given partition is processed at a time This approach means that although records from different partitions may be processed in parallel, records from the same partition are processed in sequential order. ### [](#delivery-errors)Delivery errors The order in which records are delivered may be disrupted by delivery errors and any error-handling mechanisms that start up. Redpanda Connect leans towards at-least-once delivery unless instructed otherwise, and this includes reattempting delivery of data when the ordering of that data is no longer guaranteed. For example, a batch of records is sent to an output broker and only a subset of records are delivered. In this scenario, Redpanda Connect (by default) attempts to deliver the records that failed, even though these delivery failures may have been sent before records that were delivered successfully. #### [](#use-a-fallback-output)Use a fallback output To prevent delivery errors from disrupting the order of records, you must specify a [`fallback`](../../outputs/fallback/) output in your pipeline configuration. When adding a `fallback` output, it is good practice to set the `auto_retry_nacks` field to `false`. This also improves the throughput of your pipeline. For example, the following configuration includes a `fallback` output. If Redpanda Connect fails to write delivery errors to the `foo` topic, it then attempts to write them into a dead letter queue topic (`foo_dlq`), which is retried indefinitely as a way to apply back pressure. ```yaml output: fallback: - redpanda_common: topic: foo - retry: output: redpanda_common: topic: foo_dlq ``` ## [](#batching)Batching Records are processed and delivered from each partition in the same batches as they are received from brokers. Batch sizes are dynamically sized in order to optimize throughput, but you can tune them further using the following configuration fields: - `fetch_max_partition_bytes` - `fetch_max_bytes` You can break batches down further using the [`split`](../../processors/split/) processor. ## [](#metrics)Metrics This input emits a `redpanda_lag` metric with `topic` and `partition` labels for each consumed topic. The metric records the number of produced messages that remain to be read from each topic/partition pair by the specified consumer group. ## [](#metadata)Metadata This input adds the following metadata fields to each message: - `kafka_key` - `kafka_topic` - `kafka_partition` - `kafka_offset` - `kafka_lag` - `kafka_timestamp_ms` - `kafka_timestamp_unix` - `kafka_tombstone_message` - All record headers ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay messages that are rejected (nacked) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#client_id)`client_id` An identifier for the client connection. **Type**: `string` **Default**: `redpanda-connect` ### [](#commit_period)`commit_period` The period of time between each commit of the current partition offsets. Offsets are always committed during shutdown. **Type**: `string` **Default**: `5s` ### [](#conn_idle_timeout)`conn_idle_timeout` The maximum duration that connections can remain idle before they are automatically closed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `20s` ### [](#consumer_group)`consumer_group` An optional consumer group. When this value is specified: - The partitions of any topics, specified in the `topics` field, are automatically distributed across consumers sharing a consumer group - Partition offsets are automatically committed and resumed under this name Consumer groups are not supported when you specify explicit partitions to consume from in the `topics` field. **Type**: `string` ### [](#extract_tracing_map)`extract_tracing_map` EXPERIMENTAL: A [Bloblang mapping](../../../guides/bloblang/about/) that attempts to extract an object containing tracing propagation information, which will then be used as the root tracing span for the message. The specification of the extracted fields must match the format used by the service wide tracer. Requires version 3.45.0 or later. **Type**: `string` ```yaml # Examples: extract_tracing_map: root = @ # --- extract_tracing_map: root = this.meta.span ``` ### [](#fetch_max_bytes)`fetch_max_bytes` The maximum number of bytes that a broker tries to send during a fetch. If individual records are larger than the `fetch_max_bytes` value, brokers will still send them. **Type**: `string` **Default**: `50MiB` ### [](#fetch_max_partition_bytes)`fetch_max_partition_bytes` The maximum number of bytes that are consumed from a single partition in a fetch request. This field is equivalent to the Java setting `fetch.max.partition.bytes`. If a single batch is larger than the `fetch_max_partition_bytes` value, the batch is still sent so that the client can make progress. **Type**: `string` **Default**: `1MiB` ### [](#fetch_max_wait)`fetch_max_wait` The maximum period of time a broker can wait for a fetch response to reach the required minimum number of bytes (`fetch_min_bytes`). **Type**: `string` **Default**: `5s` ### [](#fetch_min_bytes)`fetch_min_bytes` The minimum number of bytes that a broker tries to send during a fetch. This field is equivalent to the Java setting `fetch.min.bytes`. **Type**: `string` **Default**: `1B` ### [](#heartbeat_interval)`heartbeat_interval` When you specify a `consumer_group`, `heartbeat_interval` sets how frequently a consumer group member should send heartbeats to Apache Kafka. Apache Kafka uses heartbeats to make sure that a group member’s session is active. You must set `heartbeat_interval` to less than one-third of `session_timeout`. This field is equivalent to the Java `heartbeat.interval.ms` setting and accepts Go duration format strings such as `10s` or `2m`. **Type**: `string` **Default**: `3s` ### [](#instance_id)`instance_id` When you specify a [`consumer_group`](#consumer_group), assign a unique value to `instance_id` to define the group’s static membership, which can prevent unnecessary rebalances during reconnections. When you assign an instance ID, the client does not automatically leave the consumer group when it disconnects. To remove the client, you must use an external admin command on behalf of the instance ID. **Type**: `string` **Default**: `""` ### [](#max_yield_batch_bytes)`max_yield_batch_bytes` The maximum size (in bytes) for each batch yielded by this input. This value must be less than or equal to the `partition_buffer_bytes`. If using Redpanda output, this value should not be greater than the `max_message_bytes` option value (1MB by default), and for high-throughput scenarios they should be equal. **Type**: `string` **Default**: `32KB` ### [](#metadata_max_age)`metadata_max_age` The maximum period of time after which metadata is refreshed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Lower values provide more responsive topic and partition discovery but may increase broker load. Higher values reduce broker queries but can delay detection of topology changes. **Type**: `string` **Default**: `1m` ### [](#partition_buffer_bytes)`partition_buffer_bytes` A buffer size (in bytes) for each consumed partition, which allows the internal queuing of records before they are flushed. Increasing this value may improve throughput but results in higher memory utilization. Each buffer can grow slightly beyond this value. **Type**: `string` **Default**: `1MB` ### [](#rack_id)`rack_id` A rack specifies where the client is physically located, and changes fetch requests to consume from the closest replica as opposed to the leader replica. **Type**: `string` **Default**: `""` ### [](#rebalance_timeout)`rebalance_timeout` When you specify a [`consumer_group`](#consumer_group), `rebalance_timeout` sets a time limit for all consumer group members to complete their work and commit offsets after a rebalance has begun. The timeout excludes the time taken to detect a failed or late heartbeat, which indicates a rebalance is required. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `45s` ### [](#regexp_topics_exclude)`regexp_topics_exclude[]` A list of regular expression patterns for excluding topics when regex mode is enabled (using `regexp_topics_include` or the deprecated `regexp_topics` boolean). Topics matching any of these patterns will be excluded from consumption, even if they match include patterns. Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so use `^` and `$` for exact matching. Exclude patterns are applied after include patterns, providing fine-grained control over topic selection. Example: `regexp_topics_exclude: ["^_", ".**-temp$", ".**-test.*"]` excludes topics starting with underscore, ending with `-temp`, or containing `-test`. **Type**: `array` ### [](#regexp_topics_include)`regexp_topics_include[]` A list of regular expression patterns for matching topics to consume from. When specified, the client will periodically refresh the list of matching topics based on the `metadata_max_age` interval. Each pattern is a full regular expression evaluated against the complete topic name. Patterns are not anchored by default, so `logs_.` **matches `my-logs_events` and `logs_errors`. Use `^logs_.`**`$` to match only topics starting with `logs_`. This field enables regex mode (replacing the deprecated `regexp_topics` boolean) and cannot be used together with explicit `topics` lists. Use `regexp_topics_exclude` to filter out specific patterns from the matched topics. Example: `regexp_topics_include: ["events_.**", "logs_.**"]` consumes from all topics starting with `events_` or `logs_`. **Type**: `array` ```yaml # Examples: regexp_topics_include: - logs_.* - metrics_.* # --- regexp_topics_include: - "events_[0-9]+" ``` ### [](#request_timeout_overhead)`request_timeout_overhead` Grants an additional buffer or overhead to requests that have timeout fields defined. This field is based on the behavior of Apache Kafka’s `request.timeout.ms` parameter. **Type**: `string` **Default**: `10s` ### [](#sasl)`sasl[]` Specify one or more methods or mechanisms of SASL authentication. They are tried in order. If the broker supports the first SASL mechanism, all connections use it. If the first mechanism fails, the client picks the first supported mechanism. If the broker does not support any client mechanisms, all connections fail. **Type**: `object` ```yaml # Examples: sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ### [](#sasl-aws)`sasl[].aws` Contains AWS specific fields for when the `mechanism` is set to `AWS_MSK_IAM`. **Type**: `object` ### [](#sasl-aws-credentials)`sasl[].aws.credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](../../../guides/cloud/aws/). **Type**: `object` ### [](#sasl-aws-credentials-from_ec2_role)`sasl[].aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#sasl-aws-credentials-id)`sasl[].aws.credentials.id` The ID of credentials to use. **Type**: `string` ### [](#sasl-aws-credentials-profile)`sasl[].aws.credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#sasl-aws-credentials-role)`sasl[].aws.credentials.role` A role ARN to assume. **Type**: `string` ### [](#sasl-aws-credentials-role_external_id)`sasl[].aws.credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#sasl-aws-credentials-secret)`sasl[].aws.credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#sasl-aws-credentials-token)`sasl[].aws.credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#sasl-aws-endpoint)`sasl[].aws.endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#sasl-aws-region)`sasl[].aws.region` The AWS region to target. **Type**: `string` ### [](#sasl-aws-tcp)`sasl[].aws.tcp` TCP socket configuration. **Type**: `object` ### [](#sasl-aws-tcp-connect_timeout)`sasl[].aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-aws-tcp-keep_alive)`sasl[].aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#sasl-aws-tcp-keep_alive-count)`sasl[].aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#sasl-aws-tcp-keep_alive-idle)`sasl[].aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-keep_alive-interval)`sasl[].aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-tcp_user_timeout)`sasl[].aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-extensions)`sasl[].extensions` Key/value pairs to add to OAUTHBEARER authentication requests. **Type**: `string` ### [](#sasl-mechanism)`sasl[].mechanism` The SASL mechanism to use. **Type**: `string` | Option | Summary | | --- | --- | | AWS_MSK_IAM | AWS IAM based authentication as specified by the 'aws-msk-iam-auth' java library. | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. | | REDPANDA_CLOUD_SERVICE_ACCOUNT | Redpanda Cloud Service Account authentication when running in Redpanda Cloud. | | SCRAM-SHA-256 | SCRAM based authentication as specified in RFC5802. | | SCRAM-SHA-512 | SCRAM based authentication as specified in RFC5802. | | none | Disable sasl authentication | ### [](#sasl-password)`sasl[].password` A password to provide for PLAIN or SCRAM-\* authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#sasl-token)`sasl[].token` The token to use for a single session’s OAUTHBEARER authentication. **Type**: `string` **Default**: `""` ### [](#sasl-username)`sasl[].username` A username to provide for PLAIN or SCRAM-\* authentication. **Type**: `string` **Default**: `""` ### [](#seed_brokers)`seed_brokers[]` A list of broker addresses to connect to in order. Use commas to separate multiple addresses in a single list item. Optional when `seed_brokers` is configured in a top-level `redpanda` block. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" # --- seed_brokers: - "foo:9092" - "bar:9092" # --- seed_brokers: - "foo:9092,bar:9092" ``` ### [](#session_timeout)`session_timeout` When you specify a `consumer_group`, `session_timeout` sets the maximum interval between heartbeats sent by a consumer group member to the broker. If a broker doesn’t receive a heartbeat from a group member before the timeout expires, it removes the member from the consumer group and initiates a rebalance. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `1m` ### [](#start_offset)`start_offset` Specify the offset from which this input starts or restarts consuming messages. Restarts occur when the `OffsetOutOfRange` error is seen during a fetch. **Type**: `string` **Default**: `earliest` | Option | Summary | | --- | --- | | committed | Prevents consuming a partition in a group if the partition has no prior commits. Corresponds to Kafka’s auto.offset.reset=none option | | earliest | Start from the earliest offset. Corresponds to Kafka’s auto.offset.reset=earliest option. | | latest | Start from the latest offset. Corresponds to Kafka’s auto.offset.reset=latest option. | ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timely_nacks_maximum_wait)`timely_nacks_maximum_wait` EXPERIMENTAL: Specify a maximum period of time in which each message can be consumed and awaiting either acknowledgement or rejection before rejection is instead forced. This can be useful for avoiding situations where certain downstream components can result in blocked confirmation of delivery that exceeds SLAs. Accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic_lag_refresh_period)`topic_lag_refresh_period` The interval between refresh cycles. During each cycle, this input queries the Redpanda Connect server to calculate the topic lag minus the number of produced messages that remain to be read from each topic/partition pair by the specified consumer group. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `5s` ### [](#topics)`topics[]` A list of topics to consume from. Use commas to separate multiple topics in a single element. When a `consumer_group` is specified, partitions are automatically distributed across consumers of a topic. Otherwise, all partitions are consumed. Alternatively, you can specify explicit partitions to consume by using a colon after the topic name. For example, `foo:0` would consume the partition `0` of the topic foo. This syntax supports ranges. For example, `foo:0-10` would consume partitions `0` through to `10` inclusive. It is also possible to specify an explicit offset to consume from by adding another colon after the partition. For example, `foo:0:10` would consume the partition `0` of the topic `foo` starting from the offset `10`. If the offset is not present (or remains unspecified) then the field `start_offset` determines which offset to start from. **Type**: `array` ```yaml # Examples: topics: - foo - bar # --- topics: - things.* # --- topics: - "foo,bar" # --- topics: - "foo:0" - "bar:1" - "bar:3" # --- topics: - "foo:0,bar:1,bar:3" # --- topics: - "foo:0-5" ``` ### [](#transaction_isolation_level)`transaction_isolation_level` The isolation level for handling transactional messages. This setting determines how transactions are processed and affects data consistency guarantees. **Type**: `string` **Default**: `read_uncommitted` | Option | Summary | | --- | --- | | read_committed | If set, only committed transactional records are processed. | | read_uncommitted | If set, then uncommitted records are processed. | ### [](#unordered_processing)`unordered_processing` Allows consumers to process messages of any given partition in parallel, which may result in unordered processing. This option enables asynchronous publishing at the output level. The maximum parallelization of each partition is determined by the `checkpoint_limit` field. **Type**: `object` ### [](#unordered_processing-batching)`unordered_processing.batching` Allows you to configure a [batching policy](../../../configuration/batching/) that applies to individual topic partitions in order to batch messages together before flushing them for processing. Batching can be beneficial for performance and useful for windowed processing, and doing so preserves the ordering of topic partitions. **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#unordered_processing-batching-byte_size)`unordered_processing.batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#unordered_processing-batching-check)`unordered_processing.batching.check` A [Bloblang query](../../../guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#unordered_processing-batching-count)`unordered_processing.batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#unordered_processing-batching-period)`unordered_processing.batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#unordered_processing-batching-processors)`unordered_processing.batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#unordered_processing-checkpoint_limit)`unordered_processing.checkpoint_limit` Determines how many messages of the same partition can be processed in parallel before applying back pressure. When a message of a given offset is delivered to the output the offset is only allowed to be committed when all messages of prior offsets have also been delivered, this ensures at-least-once delivery guarantees. However, this mechanism also increases the likelihood of duplicates in the event of crashes or server faults, reducing the checkpoint limit will mitigate this. **Type**: `int` **Default**: `1024` ### [](#unordered_processing-enabled)`unordered_processing.enabled` Whether to enable the unordered processing of messages from a given partition. **Type**: `bool` **Default**: `false` --- # Page 93: resource **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/resource.md --- # resource --- title: resource latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/resource page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/resource.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/resource.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/resource/)[Output](/redpanda-connect/components/outputs/resource/)[Processor](/redpanda-connect/components/processors/resource/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/resource/ "View the Cloud version of this component"), Self-Managed Resource is an input type that channels messages from a resource input, identified by its name. ```yml inputs: label: "" resource: "" ``` Resources allow you to tidy up deeply nested configs. For example, the config: ```yaml input: broker: inputs: - kafka: addresses: [ TODO ] topics: [ foo ] consumer_group: foogroup - gcp_pubsub: project: bar subscription: baz ``` Could also be expressed as: ```yaml input: broker: inputs: - resource: foo - resource: bar input_resources: - label: foo kafka: addresses: [ TODO ] topics: [ foo ] consumer_group: foogroup - label: bar gcp_pubsub: project: bar subscription: baz ``` Resources also allow you to reference a single input in multiple places, such as multiple streams mode configs, or multiple entries in a broker input. However, when a resource is referenced more than once the messages it produces are distributed across those references, so each message will only be directed to a single reference, not all of them. You can find out more about resources in [Resources](../../../configuration/resources/). --- # Page 94: schema_registry **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/schema_registry.md --- # schema\_registry --- title: schema_registry latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/schema_registry page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/schema_registry.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/schema_registry.adoc page-git-created-date: "2024-09-04" page-git-modified-date: "2026-01-05" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/schema_registry/)[Output](/redpanda-connect/components/outputs/schema_registry/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/schema_registry/ "View the Cloud version of this component"), Self-Managed Reads schemas from a schema registry. You can use this connector to extract and back up schemas during a data migration. This input uses the [Franz Kafka Schema Registry client](https://github.com/twmb/franz-go/tree/master/pkg/sr). Introduced in version 4.32.2. #### Common ```yml inputs: label: "" schema_registry: url: "" # No default (required) auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" schema_registry: url: "" # No default (required) include_deleted: false subject_filter: "" fetch_in_order: true tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] auto_replay_nacks: true oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} ``` ## [](#metadata)Metadata The `schema_registry` input adds the following metadata fields to each message: ```text - schema_registry_subject - schema_registry_version ``` You can access these metadata fields using [function interpolation](../../../configuration/interpolation/#bloblang-queries). ## [](#example)Example This example reads all schemas from a schema registry that are associated with subjects matching the `^foo.*` filter, including deleted schemas. ```yaml input: schema_registry: url: http://localhost:8081 include_deleted: true subject_filter: ^foo.* ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay messages that are rejected (nacked) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#basic_auth)`basic_auth` Configure basic authentication for requests from this component to your schema registry. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` The password to use for authentication. Used together with `username` for basic authentication or with encrypted private keys for secure access. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` The username of the account credentials to authenticate as. Used together with `password` for basic authentication. **Type**: `string` **Default**: `""` ### [](#fetch_in_order)`fetch_in_order` Indicate whether to fetch all schemas from the schema registry service and sort them by ID. Set this value to `true` if you use schemas that refer to other schemas (schema references). Requires version 4.37.0 or later. **Type**: `bool` **Default**: `true` ### [](#include_deleted)`include_deleted` Include deleted entities. **Type**: `bool` **Default**: `false` ### [](#jwt)`jwt` Beta Configure JSON Web Token (JWT) authentication for secure data transmission from your schema registry to this component. This feature is in beta and may change in future releases. **Type**: `object` ### [](#jwt-claims)`jwt.claims` Values used to pass the identity of the authenticated entity to the service provider. In this case, between this component and the schema registry. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` The key/value pairs that identify the type of token and signing algorithm. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` A PEM-encoded file containing a private key that is formatted using either PKCS1 or PKCS8 standards. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` The method used to sign the token, such as RS256, RS384, RS512 or EdDSA. **Type**: `string` **Default**: `""` ### [](#oauth)`oauth` Configure OAuth version 1.0 to give this component authorized access to your schema registry. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` The value this component can use to gain access to the data in the schema registry. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` The secret that establishes ownership of the `oauth.access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` The value used to identify this component or client to your schema registry. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#subject_filter)`subject_filter` Include only subjects which match the regular expression filter, or leave this field value blank to select all subjects. **Type**: `string` **Default**: `""` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The base URL of the schema registry service. **Type**: `string` --- # Page 95: sequence **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/sequence.md --- # sequence --- title: sequence latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/sequence page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/sequence.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/sequence.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/sequence/ "View the Cloud version of this component"), Self-Managed Reads messages from a sequence of child inputs, starting with the first and once that input gracefully terminates starts consuming from the next, and so on. #### Common ```yml inputs: label: "" sequence: inputs: [] # No default (required) ``` #### Advanced ```yml inputs: label: "" sequence: sharded_join: type: none id_path: "" iterations: 1 merge_strategy: array inputs: [] # No default (required) ``` This input is useful for consuming from inputs that have an explicit end but must not be consumed in parallel. ## [](#examples)Examples ### [](#end-of-stream-message)End of Stream Message A common use case for sequence might be to generate a message at the end of our main input. With the following config once the records within `./dataset.csv` are exhausted our final payload `{"status":"finished"}` will be routed through the pipeline. ```yaml input: sequence: inputs: - file: paths: [ ./dataset.csv ] scanner: csv: {} - generate: count: 1 mapping: 'root = {"status":"finished"}' ``` ### [](#joining-data-simple)Joining Data (Simple) Redpanda Connect can be used to join unordered data from fragmented datasets in memory by specifying a common identifier field and a number of sharded iterations. For example, given two CSV files, the first called "main.csv", which contains rows of user data: ```csv uuid,name,age AAA,Melanie,34 BBB,Emma,28 CCC,Geri,45 ``` And the second called "hobbies.csv" that, for each user, contains zero or more rows of hobbies: ```csv uuid,hobby CCC,pokemon go AAA,rowing AAA,golf ``` We can parse and join this data into a single dataset: ```json {"uuid":"AAA","name":"Melanie","age":34,"hobbies":["rowing","golf"]} {"uuid":"BBB","name":"Emma","age":28} {"uuid":"CCC","name":"Geri","age":45,"hobbies":["pokemon go"]} ``` With the following config: ```yaml input: sequence: sharded_join: type: full-outer id_path: uuid merge_strategy: array inputs: - file: paths: - ./hobbies.csv - ./main.csv scanner: csv: {} ``` ### [](#joining-data-advanced)Joining Data (Advanced) In this example we are able to join unordered and fragmented data from a combination of CSV files and newline-delimited JSON documents by specifying multiple sequence inputs with their own processors for extracting the structured data. The first file "main.csv" contains straight forward CSV data: ```csv uuid,name,age AAA,Melanie,34 BBB,Emma,28 CCC,Geri,45 ``` And the second file called "hobbies.ndjson" contains JSON documents, one per line, that associate an identifier with an array of hobbies. However, these data objects are in a nested format: ```json {"document":{"uuid":"CCC","hobbies":[{"type":"pokemon go"}]}} {"document":{"uuid":"AAA","hobbies":[{"type":"rowing"},{"type":"golf"}]}} ``` And so we will want to map these into a flattened structure before the join, and then we will end up with a single dataset that looks like this: ```json {"uuid":"AAA","name":"Melanie","age":34,"hobbies":["rowing","golf"]} {"uuid":"BBB","name":"Emma","age":28} {"uuid":"CCC","name":"Geri","age":45,"hobbies":["pokemon go"]} ``` With the following config: ```yaml input: sequence: sharded_join: type: full-outer id_path: uuid iterations: 10 merge_strategy: array inputs: - file: paths: [ ./main.csv ] scanner: csv: {} - file: paths: [ ./hobbies.ndjson ] scanner: lines: {} processors: - mapping: | root.uuid = this.document.uuid root.hobbies = this.document.hobbies.map_each(this.type) ``` ## [](#fields)Fields ### [](#inputs)`inputs[]` An array of inputs to read from sequentially. **Type**: `input` ### [](#sharded_join)`sharded_join` EXPERIMENTAL: Provides a way to perform outer joins of arbitrarily structured and unordered data resulting from the input sequence, even when the overall size of the data surpasses the memory available on the machine. When configured the sequence of inputs will be consumed one or more times according to the number of iterations, and when more than one iteration is specified each iteration will process an entirely different set of messages by sharding them by the ID field. Increasing the number of iterations reduces the memory consumption at the cost of needing to fully parse the data each time. Each message must be structured (JSON or otherwise processed into a structured form) and the fields will be aggregated with those of other messages sharing the ID. At the end of each iteration the joined messages are flushed downstream before the next iteration begins, hence keeping memory usage limited. Requires version 3.40.0 or later. **Type**: `object` ### [](#sharded_join-id_path)`sharded_join.id_path` A [dot path](../../../configuration/field_paths/) that points to a common field within messages of each fragmented data set and can be used to join them. Messages that are not structured or are missing this field will be dropped. This field must be set in order to enable joins. **Type**: `string` **Default**: `""` ### [](#sharded_join-iterations)`sharded_join.iterations` The total number of iterations (shards), increasing this number will increase the overall time taken to process the data, but reduces the memory used in the process. The real memory usage required is significantly higher than the real size of the data and therefore the number of iterations should be at least an order of magnitude higher than the available memory divided by the overall size of the dataset. **Type**: `int` **Default**: `1` ### [](#sharded_join-merge_strategy)`sharded_join.merge_strategy` The chosen strategy to use when a data join would otherwise result in a collision of field values. The strategy `array` means non-array colliding values are placed into an array and colliding arrays are merged. The strategy `replace` replaces old values with new values. The strategy `keep` keeps the old value. **Type**: `string` **Default**: `array` **Options**: `array`, `replace`, `keep` ### [](#sharded_join-type)`sharded_join.type` The type of join to perform. A `full-outer` ensures that all identifiers seen in any of the input sequences are sent, and is performed by consuming all input sequences before flushing the joined results. An `outer` join consumes all input sequences but only writes data joined from the last input in the sequence, similar to a left or right outer join. With an `outer` join if an identifier appears multiple times within the final sequence input it will be flushed each time it appears. `full-outter` and `outter` have been deprecated in favour of `full-outer` and `outer`. **Type**: `string` **Default**: `none` **Options**: `none`, `full-outer`, `outer`, `full-outter`, `outter` --- # Page 96: sftp **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/sftp.md --- # sftp --- title: sftp latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/sftp page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/sftp.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/sftp.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/sftp/)[Output](/redpanda-connect/components/outputs/sftp/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/sftp/ "View the Cloud version of this component"), Self-Managed Consumes files from an SFTP server. Introduced in version 3.39.0. #### Common ```yml inputs: label: "" sftp: address: "" # No default (required) credentials: username: "" password: "" host_public_key_file: "" # No default (optional) host_public_key: "" # No default (optional) private_key_file: "" # No default (optional) private_key: "" # No default (optional) private_key_pass: "" paths: [] # No default (required) auto_replay_nacks: true scanner: to_the_end: {} watcher: enabled: false minimum_age: 1s poll_interval: 1s cache: "" ``` #### Advanced ```yml inputs: label: "" sftp: address: "" # No default (required) connection_timeout: 30s credentials: username: "" password: "" host_public_key_file: "" # No default (optional) host_public_key: "" # No default (optional) private_key_file: "" # No default (optional) private_key: "" # No default (optional) private_key_pass: "" max_sftp_sessions: 10 paths: [] # No default (required) auto_replay_nacks: true scanner: to_the_end: {} delete_on_finish: false watcher: enabled: false minimum_age: 1s poll_interval: 1s cache: "" ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: - sftp\_path You can access these metadata fields using [function interpolation](../../../configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#address)`address` The address (hostname or IP address) of the SFTP server to connect to. **Type**: `string` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#connection_timeout)`connection_timeout` The connection timeout to use when connecting to the target server. **Type**: `string` **Default**: `30s` ### [](#credentials)`credentials` The credentials required to log in to the SFTP server. This can include a username and password, or a private key for secure access. **Type**: `object` ### [](#credentials-host_public_key)`credentials.host_public_key` The raw contents of the SFTP server’s public key, used for host key verification. **Type**: `string` ### [](#credentials-host_public_key_file)`credentials.host_public_key_file` The path to the SFTP server’s public key file, used for host key verification. **Type**: `string` ### [](#credentials-password)`credentials.password` The password to use for authentication. Used together with `username` for basic authentication or with encrypted private keys for secure access. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#credentials-private_key)`credentials.private_key` The private key used to authenticate with the SFTP server. This field provides an alternative to the [`private_key_file`](#credentials-private_key_file). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#credentials-private_key_file)`credentials.private_key_file` The path to a private key file used to authenticate with the SFTP server. You can also provide a private key using the [`private_key`](#credentials-private_key) field. **Type**: `string` ### [](#credentials-private_key_pass)`credentials.private_key_pass` A passphrase for the private key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#credentials-username)`credentials.username` The username required to authenticate with the SFTP server. **Type**: `string` **Default**: `""` ### [](#delete_on_finish)`delete_on_finish` Whether to delete files from the server once they are processed. **Type**: `bool` **Default**: `false` ### [](#max_sftp_sessions)`max_sftp_sessions` The maximum number of SFTP sessions. **Type**: `int` **Default**: `10` ### [](#paths)`paths[]` A list of paths to consume sequentially. Glob patterns are supported. **Type**: `array` ### [](#scanner)`scanner` The [scanner](../../scanners/about/) by which the stream of bytes consumed will be broken out into individual messages. Scanners are useful for processing large sources of data without holding the entirety of it within memory. For example, the `csv` scanner allows you to process individual CSV rows without loading the entire CSV file in memory at once. Requires version 4.25.0 or later. **Type**: `scanner` **Default**: ```yaml to_the_end: {} ``` ### [](#watcher)`watcher` An experimental mode whereby the input will periodically scan the target paths for new files and consume them, when all files are consumed the input will continue polling for new files. Requires version 3.42.0 or later. **Type**: `object` ### [](#watcher-cache)`watcher.cache` A [cache resource](../../caches/about/) for storing the paths of files already consumed. **Type**: `string` **Default**: `""` ### [](#watcher-enabled)`watcher.enabled` Whether file watching is enabled. **Type**: `bool` **Default**: `false` ### [](#watcher-minimum_age)`watcher.minimum_age` The minimum period of time since a file was last updated before attempting to consume it. Increasing this period decreases the likelihood that a file will be consumed whilst it is still being written to. **Type**: `string` **Default**: `1s` ```yaml # Examples: minimum_age: 10s # --- minimum_age: 1m # --- minimum_age: 10m ``` ### [](#watcher-poll_interval)`watcher.poll_interval` The interval between each attempt to scan the target paths for new files. **Type**: `string` **Default**: `1s` ```yaml # Examples: poll_interval: 100ms # --- poll_interval: 1s ``` --- # Page 97: slack_users **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/slack_users.md --- # slack\_users --- title: slack_users latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/slack_users page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/slack_users.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/slack_users.adoc page-git-created-date: "2025-05-02" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/slack_users/ "View the Cloud version of this component"), Self-Managed **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Returns [the full profile](https://api.slack.com/methods/users.list#examples) of all users in your Slack organization using the API method [users.list](https://api.slack.com/methods/users.list). Optionally, you can filter the list of returned users by team ID. This input is useful when you need to: - Join user information to Slack posts. - Ingest user information into a data lakehouse to create joins with other fields. Introduced in version 4.52.0. ```yml inputs: label: "" slack_users: bot_token: "" # No default (required) team_id: "" auto_replay_nacks: true ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay messages that are rejected (nacked) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#bot_token)`bot_token` Your [Slack bot user’s OAuth token](https://api.slack.com/concepts/token-types), which must have the [`users.read` scope](https://api.slack.com/scopes/users:read) to access your Slack organization. **Type**: `string` ### [](#team_id)`team_id` The encoded ID of a Slack team by which to filter the list of returned users, which you can get from the [`team.info` Slack API method](https://api.slack.com/methods/team.info). If `team_id` is left empty, users from all teams within the organization are returned. **Type**: `string` **Default**: `""` --- # Page 98: slack **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/slack.md --- # slack --- title: slack latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/slack page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/slack.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/slack.adoc page-git-created-date: "2025-05-02" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/slack/ "View the Cloud version of this component"), Self-Managed **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Connects to Slack using [Socket Mode](https://api.slack.com/apis/socket-mode), and can receive events, interactions (automated and user-initiated), and slash commands. This input is useful for: - Building bots that can query or write data. - Sending events to data warehouses. You could also try pairing this input with Redpanda Connect’s AI processors, which use the prefixes `cohere`, `openai`, and `ollama`. Introduced in version 4.51.0. ```yml inputs: label: "" slack: app_token: "" # No default (required) bot_token: "" # No default (required) auto_replay_nacks: true ``` See also: [Examples](#examples) ## [](#metadata)Metadata Each message emitted from this input has an `@type` metadata flag to indicate the event type, either `"events_api"`, `"interactions"`, or `"slash_commands"`. ## [](#fields)Fields ### [](#app_token)`app_token` The app-level token to use to authenticate and connect to Slack. **Type**: `string` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether to automatically replay messages that are rejected (nacked) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure. Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams, as the original shape of the data is discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#bot_token)`bot_token` Your Slack bot user’s OAuth token, which must have the [`connections.write` scope](https://api.slack.com/scopes/connections:write) to access your Slack app’s [Socket Mode WebSocket URL](https://api.slack.com/methods/apps.connections.open). **Type**: `string` ## [](#examples)Examples ### [](#echo-slackbot)Echo Slackbot A slackbot that echo messages from other users ```yaml input: slack: app_token: "${APP_TOKEN:xapp-demo}" bot_token: "${BOT_TOKEN:xoxb-demo}" pipeline: processors: - mutation: | # ignore hidden or non message events if this.event.type != "message" || (this.event.hidden | false) { root = deleted() } # Don't respond to our own messages if this.authorizations.any(auth -> auth.user_id == this.event.user) { root = deleted() } output: slack_post: bot_token: "${BOT_TOKEN:xoxb-demo}" channel_id: "${!this.event.channel}" thread_ts: "${!this.event.ts}" text: "ECHO: ${!this.event.text}" ``` --- # Page 99: socket_server **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/socket_server.md --- # socket\_server --- title: socket_server latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/socket_server page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/socket_server.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/socket_server.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** Self-Managed Creates a server that receives a stream of messages over a TCP, UDP or Unix socket. ```yml inputs: label: "" socket_server: network: "" # No default (required) address: "" # No default (required) address_cache: "" # No default (optional) tls: cert_file: "" # No default (optional) key_file: "" # No default (optional) self_signed: false client_auth: no auto_replay_nacks: true scanner: lines: {} ``` ## [](#fields)Fields ### [](#address)`address` The address to listen from. **Type**: `string` ```yaml # Examples: address: /tmp/benthos.sock # --- address: 0.0.0.0:6000 ``` ### [](#address_cache)`address_cache` An optional [`cache`](../../caches/about/) within which this input should write it’s bound address once known. The key of the cache item containing the address will be the label of the component suffixed with `_address` (e.g. `foo_address`), or `socket_server_address` when a label has not been provided. This is useful in situations where the address is dynamically allocated by the server (`127.0.0.1:0`) and you want to store the allocated address somewhere for reference by other systems and components. Requires version 4.25.0 or later. **Type**: `string` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#network)`network` A network type to accept. **Type**: `string` **Options**: `unix`, `tcp`, `udp`, `tls`, `unixgram` ### [](#scanner)`scanner` The [scanner](../../scanners/about/) by which the stream of bytes consumed will be broken out into individual messages. Scanners are useful for processing large sources of data without holding the entirety of it within memory. For example, the `csv` scanner allows you to process individual CSV rows without loading the entire CSV file in memory at once. Requires version 4.25.0 or later. **Type**: `scanner` **Default**: ```yaml lines: {} ``` ### [](#tcp)`tcp` TCP listener socket configuration. **Type**: `object` ### [](#tcp-reuse_addr)`tcp.reuse_addr` Enable SO\_REUSEADDR, allowing binding to ports in TIME\_WAIT state. Useful for graceful restarts and config reloads where the server needs to rebind to the same port immediately after shutdown. **Type**: `bool` **Default**: `false` ### [](#tcp-reuse_port)`tcp.reuse_port` Enable SO\_REUSEPORT, allowing multiple sockets to bind to the same port for load balancing across multiple processes/threads. **Type**: `bool` **Default**: `false` ### [](#tls)`tls` TLS specific configuration, valid when the `network` is set to `tls`. **Type**: `object` ### [](#tls-cert_file)`tls.cert_file` PEM encoded certificate for use with TLS. **Type**: `string` ### [](#tls-client_auth)`tls.client_auth` Specifies how client authentication is handled when using TLS. Requires version 4.44.1 or later. **Type**: `string` **Default**: `no` | Option | Summary | | --- | --- | | no | client certificate is not requested nor required. | | request | will request client certificate, not require it. | | require_any | will accept any client certificate, even if not valid. | | require_valid | requires a valid client certificate. | | verify_if_given | will verify a certificate, if one is sent by the client. | ### [](#tls-key_file)`tls.key_file` PEM encoded private key for use with TLS. **Type**: `string` ### [](#tls-self_signed)`tls.self_signed` Whether to generate self signed certificates. **Type**: `bool` **Default**: `false` --- # Page 100: socket **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/socket.md --- # socket --- title: socket latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/socket page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/socket.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/socket.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/socket/)[Output](/redpanda-connect/components/outputs/socket/) **Available in:** Self-Managed Connects to a tcp or unix socket and consumes a continuous stream of messages. ```yml inputs: label: "" socket: network: "" # No default (required) address: "" # No default (required) auto_replay_nacks: true open_message_mapping: "" # No default (optional) scanner: lines: {} ``` ## [](#fields)Fields ### [](#address)`address` The address to connect to. **Type**: `string` ```yaml # Examples: address: /tmp/benthos.sock # --- address: 127.0.0.1:6000 ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#network)`network` A network type to assume (unix|tcp). **Type**: `string` **Options**: `unix`, `tcp` ### [](#open_message_mapping)`open_message_mapping` An optional [Bloblang mapping](../../../guides/bloblang/about/) which should evaluate to a string which will be sent upstream before the downstream data flow starts. **Type**: `string` ```yaml # Examples: open_message_mapping: root = "username,password" ``` ### [](#scanner)`scanner` The [scanner](../../scanners/about/) by which the stream of bytes consumed will be broken out into individual messages. Scanners are useful for processing large sources of data without holding the entirety of it within memory. For example, the `csv` scanner allows you to process individual CSV rows without loading the entire CSV file in memory at once. Requires version 4.25.0 or later. **Type**: `scanner` **Default**: ```yaml lines: {} ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` --- # Page 101: spicedb_watch **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/spicedb_watch.md --- # spicedb\_watch --- title: spicedb_watch latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/spicedb_watch page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/spicedb_watch.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/spicedb_watch.adoc page-git-created-date: "2024-11-19" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/spicedb_watch/ "View the Cloud version of this component"), Self-Managed Consumes messages from the [Watch API](https://buf.build/authzed/api/docs/main:authzed.api.v1#authzed.api.v1.WatchService.Watch) of a [SpiceDB](https://authzed.com/docs/spicedb/getting-started/discovering-spicedb) instance. This input is useful if you have downstream applications that need to react to real-time changes in data managed by SpiceDB. Introduced in version 4.39.0. #### Common ```yml inputs: label: "" spicedb_watch: endpoint: "" # No default (required) bearer_token: "" cache: "" # No default (required) ``` #### Advanced ```yml inputs: label: "" spicedb_watch: endpoint: "" # No default (required) bearer_token: "" max_receive_message_bytes: 4MB cache: "" # No default (required) cache_key: authzed.com/spicedb/watch/last_zed_token tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] ``` ## [](#authentication)Authentication For this input to authenticate with your SpiceDB instance, you must provide: - The [`endpoint`](#endpoint) of the SpiceDB instance - A [bearer token](#bearer_token) ## [](#configure-a-cache)Configure a cache You must use a cache resource to store the [ZedToken](https://authzed.com/docs/spicedb/concepts/consistency#zedtokens) (ID) of the latest message consumed and acknowledged by this input. Ideally, the cache should persist across restarts. This means that every time the input is initialized, it starts reading from the newest data updates. The following example uses a [`redis` cache](../../rate_limits/redis/). ```yml # Example input: label: "" spicedb_watch: endpoint: grpc.authzed.com:443 bearer_token: "" cache: "spicedb_cache" cache_resources: - label: "spicedb_cache" redis: url: redis://:6379 ``` To learn more about cache configuration, see [Resources](../../../configuration/resources/) and the [Caches section](../../caches/about/), which includes a range of cache components. ## [](#fields)Fields ### [](#bearer_token)`bearer_token` The SpiceDB bearer token to use to authenticate with your SpiceDB instance. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: bearer_token: t_your_token_here_1234567deadbeef ``` ### [](#cache)`cache` The [cache resource](#configure-a-cache) that you must configure to store the ZedToken (ID) of the last message processed. The ZedToken is stored in the cache within the `ACK` function of the message. This means that a ZedToken is only stored when a message is successfully routed through all processors and outputs in the data pipeline. **Type**: `string` ### [](#cache_key)`cache_key` The key identifier to use when storing the ZedToken (ID) of the last message received. **Type**: `string` **Default**: `authzed.com/spicedb/watch/last_zed_token` ### [](#endpoint)`endpoint` The endpoint of your SpiceDB instance. **Type**: `string` ```yaml # Examples: endpoint: grpc.authzed.com:443 ``` ### [](#max_receive_message_bytes)`max_receive_message_bytes` The maximum message size (in bytes) this input can receive. If a message exceeds this limit, an `rpc error` is written to the Redpanda Connect logs. **Type**: `string` **Default**: `4MB` ```yaml # Examples: max_receive_message_bytes: 100MB # --- max_receive_message_bytes: 50mib ``` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` --- # Page 102: splunk **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/splunk.md --- # splunk --- title: splunk latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/splunk page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/splunk.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/splunk.adoc page-git-created-date: "2024-08-15" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/splunk/ "View the Cloud version of this component"), Self-Managed **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Consumes messages from Splunk. Introduced in version 4.30.0. #### Common ```yml inputs: label: "" splunk: url: "" # No default (required) user: "" # No default (required) password: "" # No default (required) query: "" # No default (required) auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" splunk: url: "" # No default (required) user: "" # No default (required) password: "" # No default (required) query: "" # No default (required) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] auto_replay_nacks: true ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#password)`password` Splunk account password. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#query)`query` Splunk search query. **Type**: `string` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` Full HTTP Search API endpoint URL. **Type**: `string` ```yaml # Examples: url: https://foobar.splunkcloud.com/services/search/v2/jobs/export ``` ### [](#user)`user` Splunk account user. **Type**: `string` --- # Page 103: sql_raw **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/sql_raw.md --- # sql\_raw --- title: sql_raw latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/sql_raw page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/sql_raw.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/sql_raw.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/sql_raw/)[Output](/redpanda-connect/components/outputs/sql_raw/)[Processor](/redpanda-connect/components/processors/sql_raw/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/sql_raw/ "View the Cloud version of this component"), Self-Managed Executes a select query and creates a message for each row received. Introduced in version 4.10.0. #### Common ```yml inputs: label: "" sql_raw: driver: "" # No default (required) dsn: "" # No default (required) query: "" # No default (required) args_mapping: "" # No default (optional) auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" sql_raw: driver: "" # No default (required) dsn: "" # No default (required) query: "" # No default (required) args_mapping: "" # No default (optional) auto_replay_nacks: true init_files: [] # No default (optional) init_statement: "" # No default (optional) conn_max_idle_time: "" # No default (optional) conn_max_life_time: "" # No default (optional) conn_max_idle: 2 conn_max_open: "" # No default (optional) ``` When the rows from the query are exhausted, this input shuts down, allowing the pipeline to gracefully terminate or for the next input in a [sequence](../sequence/) to execute. ## [](#examples)Examples ### [](#consumes-an-sql-table-using-a-query-as-an-input)Consumes an SQL table using a query as an input. Here we perform an aggregate over a list of names in a table that are less than 3600 seconds old. ```yaml input: sql_raw: driver: postgres dsn: postgres://foouser:foopass@localhost:5432/testdb?sslmode=disable query: "SELECT name, count(*) FROM person WHERE last_updated < $1 GROUP BY name;" args_mapping: | root = [ now().ts_unix() - 3600 ] ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` An optional [Bloblang mapping](../../../guides/bloblang/about/) that includes the same number of values in an array as the placeholder arguments in the [`query`](#query) field. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#conn_max_idle)`conn_max_idle` An optional maximum number of connections in the idle connection pool. If conn\_max\_open is greater than 0 but less than the new conn\_max\_idle, then the new conn\_max\_idle will be reduced to match the conn\_max\_open limit. If `value ⇐ 0`, no idle connections are retained. The default max idle connections is currently 2. This may change in a future release. **Type**: `int` **Default**: `2` ### [](#conn_max_idle_time)`conn_max_idle_time` An optional maximum amount of time a connection may be idle. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections idle time. **Type**: `string` ### [](#conn_max_life_time)`conn_max_life_time` An optional maximum amount of time a connection may be reused. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections age. **Type**: `string` ### [](#conn_max_open)`conn_max_open` An optional maximum number of open connections to the database. If conn\_max\_idle is greater than 0 and the new conn\_max\_open is less than conn\_max\_idle, then conn\_max\_idle will be reduced to match the new conn\_max\_open limit. If `value ⇐ 0`, then there is no limit on the number of open connections. The default is 0 (unlimited). **Type**: `int` ### [](#driver)`driver` A database [driver](#drivers) to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks` ### [](#dsn)`dsn` A Data Source Name to identify the target database. #### [](#drivers)Drivers The following is a list of supported drivers, their placeholder style, and their respective DSN formats: | Driver | Data Source Name Format | | --- | --- | | clickhouse | clickhouse://[username[:password]@][netloc][:port]/dbname[?param1=value1&…​¶mN=valueN] | | mysql | [username[:password]@][protocol[(address)]]/dbname[?param1=value1&…​¶mN=valueN] | | postgres and pgx | postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&…​] | | mssql | sqlserver://[user[:password]@][netloc][:port][?database=dbname¶m1=value1&…​] | | sqlite | file:/path/to/filename.db[?param&=value1&…​] | | oracle | oracle://[username[:password]@][netloc][:port]/service_name?server=server2&server=server3 | | snowflake | username[:password]@account_identifier/dbname/schemaname[?param1=value&…​¶mN=valueN] | | trino | http[s]://user[:pass]@host[:port][?parameters] | | gocosmos | AccountEndpoint=;AccountKey=[;TimeoutMs=][;Version=][;DefaultDb/Db=][;AutoId=][;InsecureSkipVerify=] | | spanner | projects/[PROJECT]/instances/[INSTANCE]/databases/[DATABASE] | | databricks | token:@:/ | Please note that the `postgres` and `pgx` drivers enforce SSL by default, you can override this with the parameter `sslmode=disable` if required. The `pgx` driver is an alternative to the standard `postgres` (pq) driver and comes with extra functionality such as support for array insertion. The `snowflake` driver supports multiple DSN formats. Please consult [the docs](https://pkg.go.dev/github.com/snowflakedb/gosnowflake#hdr-Connection_String) for more details. For [key pair authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication), the DSN has the following format: `@//?warehouse=&role=&authenticator=snowflake_jwt&privateKey=`, where the value for the `privateKey` parameter can be constructed from an unencrypted RSA private key file `rsa_key.p8` using `openssl enc -d -base64 -in rsa_key.p8 | basenc --base64url -w0` (you can use `gbasenc` instead of `basenc` on OSX if you install `coreutils` via Homebrew). If you have a password-encrypted private key, you can decrypt it using `openssl pkcs8 -in rsa_key_encrypted.p8 -out rsa_key.p8`. Also, make sure fields such as the username are URL-encoded. The [`gocosmos`](https://pkg.go.dev/github.com/microsoft/gocosmos) driver is still experimental, but it has support for [hierarchical partition keys](https://learn.microsoft.com/en-us/azure/cosmos-db/hierarchical-partition-keys) as well as [cross-partition queries](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-query-container#cross-partition-query). Please refer to the [SQL notes](https://github.com/microsoft/gocosmos/blob/main/SQL.md) for details. **Type**: `string` ```yaml # Examples: dsn: clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60 # --- dsn: foouser:foopassword@tcp(localhost:3306)/foodb # --- dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable # --- dsn: oracle://foouser:foopass@localhost:1521/service_name # --- dsn: token:dapi1234567890ab@dbc-a1b2345c-d6e7.cloud.databricks.com:443/sql/1.0/warehouses/abc123def456 ``` ### [](#init_files)`init_files[]` An optional list of file paths containing SQL statements to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Glob patterns are supported, including super globs (double star). Care should be taken to ensure that the statements are idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If a statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `array` ```yaml # Examples: init_files: - ./init/*.sql # --- init_files: - ./foo.sql - ./bar.sql ``` ### [](#init_statement)`init_statement` An optional SQL statement to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Care should be taken to ensure that the statement is idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If the statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `string` ```yaml # Examples: init_statement: |- CREATE TABLE IF NOT EXISTS some_table ( foo varchar(50) not null, bar integer, baz varchar(50), primary key (foo) ) WITHOUT ROWID; ``` ### [](#query)`query` The query to execute. The style of placeholder to use depends on the driver, some drivers require question marks (`?`) whereas others expect incrementing dollar signs (`$1`, `$2`, and so on) or colons (`:1`, `:2` and so on). The style to use is outlined in this table: | Driver | Placeholder Style | | --- | --- | | clickhouse | Dollar sign ($) | | gocosmos | Colon (:) | | mysql | Question mark (?) | | mssql | Question mark (?) | | oracle | Colon (:) | | postgres | Dollar sign ($) | | snowflake | Question mark (?) | | spanner | Question mark (?) | | sqlite | Question mark (?) | | trino | Question mark (?) | **Type**: `string` ```yaml # Examples: query: SELECT * FROM footable WHERE user_id = $1; ``` --- # Page 104: sql_select **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/sql_select.md --- # sql\_select --- title: sql_select latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/sql_select page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/sql_select.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/sql_select.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/sql_select/)[Processor](/redpanda-connect/components/processors/sql_select/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/sql_select/ "View the Cloud version of this component"), Self-Managed Executes a select query and creates a message for each row received. Introduced in version 3.59.0. #### Common ```yml inputs: label: "" sql_select: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) columns: [] # No default (required) where: "" # No default (optional) args_mapping: "" # No default (optional) auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" sql_select: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) columns: [] # No default (required) where: "" # No default (optional) args_mapping: "" # No default (optional) prefix: "" # No default (optional) suffix: "" # No default (optional) auto_replay_nacks: true init_files: [] # No default (optional) init_statement: "" # No default (optional) conn_max_idle_time: "" # No default (optional) conn_max_life_time: "" # No default (optional) conn_max_idle: 2 conn_max_open: "" # No default (optional) ``` Once the rows from the query are exhausted this input shuts down, allowing the pipeline to gracefully terminate (or the next input in a [sequence](../sequence/) to execute). ## [](#examples)Examples ### [](#consume-a-table-postgresql)Consume a Table (PostgreSQL) Here we define a pipeline that will consume all rows from a table created within the last hour by comparing the unix timestamp stored in the row column "created\_at": ```yaml input: sql_select: driver: postgres dsn: postgres://foouser:foopass@localhost:5432/testdb?sslmode=disable table: footable columns: [ '*' ] where: created_at >= ? args_mapping: | root = [ now().ts_unix() - 3600 ] ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` An optional [Bloblang mapping](../../../guides/bloblang/about/) which should evaluate to an array of values matching in size to the number of placeholder arguments in the field `where`. **Type**: `string` ```yaml # Examples: args_mapping: root = [ "article", now().ts_format("2006-01-02") ] ``` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#columns)`columns[]` A list of columns to select. **Type**: `array` ```yaml # Examples: columns: - "*" # --- columns: - foo - bar - baz ``` ### [](#conn_max_idle)`conn_max_idle` An optional maximum number of connections in the idle connection pool. If conn\_max\_open is greater than 0 but less than the new conn\_max\_idle, then the new conn\_max\_idle will be reduced to match the conn\_max\_open limit. If `value ⇐ 0`, no idle connections are retained. The default max idle connections is currently 2. This may change in a future release. **Type**: `int` **Default**: `2` ### [](#conn_max_idle_time)`conn_max_idle_time` An optional maximum amount of time a connection may be idle. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections idle time. **Type**: `string` ### [](#conn_max_life_time)`conn_max_life_time` An optional maximum amount of time a connection may be reused. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections age. **Type**: `string` ### [](#conn_max_open)`conn_max_open` An optional maximum number of open connections to the database. If conn\_max\_idle is greater than 0 and the new conn\_max\_open is less than conn\_max\_idle, then conn\_max\_idle will be reduced to match the new conn\_max\_open limit. If `value ⇐ 0`, then there is no limit on the number of open connections. The default is 0 (unlimited). **Type**: `int` ### [](#driver)`driver` A database [driver](#drivers) to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks` ### [](#dsn)`dsn` A Data Source Name to identify the target database. #### [](#drivers)Drivers The following is a list of supported drivers, their placeholder style, and their respective DSN formats: | Driver | Data Source Name Format | | --- | --- | | clickhouse | clickhouse://[username[:password]@][netloc][:port]/dbname[?param1=value1&…​¶mN=valueN] | | mysql | [username[:password]@][protocol[(address)]]/dbname[?param1=value1&…​¶mN=valueN] | | postgres and pgx | postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&…​] | | mssql | sqlserver://[user[:password]@][netloc][:port][?database=dbname¶m1=value1&…​] | | sqlite | file:/path/to/filename.db[?param&=value1&…​] | | oracle | oracle://[username[:password]@][netloc][:port]/service_name?server=server2&server=server3 | | snowflake | username[:password]@account_identifier/dbname/schemaname[?param1=value&…​¶mN=valueN] | | trino | http[s]://user[:pass]@host[:port][?parameters] | | gocosmos | AccountEndpoint=;AccountKey=[;TimeoutMs=][;Version=][;DefaultDb/Db=][;AutoId=][;InsecureSkipVerify=] | | spanner | projects/[PROJECT]/instances/[INSTANCE]/databases/[DATABASE] | | databricks | token:@:/ | Please note that the `postgres` and `pgx` drivers enforce SSL by default, you can override this with the parameter `sslmode=disable` if required. The `pgx` driver is an alternative to the standard `postgres` (pq) driver and comes with extra functionality such as support for array insertion. The `snowflake` driver supports multiple DSN formats. Please consult [the docs](https://pkg.go.dev/github.com/snowflakedb/gosnowflake#hdr-Connection_String) for more details. For [key pair authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication), the DSN has the following format: `@//?warehouse=&role=&authenticator=snowflake_jwt&privateKey=`, where the value for the `privateKey` parameter can be constructed from an unencrypted RSA private key file `rsa_key.p8` using `openssl enc -d -base64 -in rsa_key.p8 | basenc --base64url -w0` (you can use `gbasenc` instead of `basenc` on OSX if you install `coreutils` via Homebrew). If you have a password-encrypted private key, you can decrypt it using `openssl pkcs8 -in rsa_key_encrypted.p8 -out rsa_key.p8`. Also, make sure fields such as the username are URL-encoded. The [`gocosmos`](https://pkg.go.dev/github.com/microsoft/gocosmos) driver is still experimental, but it has support for [hierarchical partition keys](https://learn.microsoft.com/en-us/azure/cosmos-db/hierarchical-partition-keys) as well as [cross-partition queries](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-query-container#cross-partition-query). Please refer to the [SQL notes](https://github.com/microsoft/gocosmos/blob/main/SQL.md) for details. **Type**: `string` ```yaml # Examples: dsn: clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60 # --- dsn: foouser:foopassword@tcp(localhost:3306)/foodb # --- dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable # --- dsn: oracle://foouser:foopass@localhost:1521/service_name # --- dsn: token:dapi1234567890ab@dbc-a1b2345c-d6e7.cloud.databricks.com:443/sql/1.0/warehouses/abc123def456 ``` ### [](#init_files)`init_files[]` An optional list of file paths containing SQL statements to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Glob patterns are supported, including super globs (double star). Care should be taken to ensure that the statements are idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If a statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `array` ```yaml # Examples: init_files: - ./init/*.sql # --- init_files: - ./foo.sql - ./bar.sql ``` ### [](#init_statement)`init_statement` An optional SQL statement to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Care should be taken to ensure that the statement is idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If the statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `string` ```yaml # Examples: init_statement: |- CREATE TABLE IF NOT EXISTS some_table ( foo varchar(50) not null, bar integer, baz varchar(50), primary key (foo) ) WITHOUT ROWID; ``` ### [](#prefix)`prefix` An optional prefix to prepend to the select query (before SELECT). **Type**: `string` ### [](#suffix)`suffix` An optional suffix to append to the select query. **Type**: `string` ### [](#table)`table` The table to select from. **Type**: `string` ```yaml # Examples: table: foo ``` ### [](#where)`where` An optional where clause to add. Placeholder arguments are populated with the `args_mapping` field. Placeholders should always be question marks, and will automatically be converted to dollar syntax when the postgres or clickhouse drivers are used. **Type**: `string` ```yaml # Examples: where: type = ? and created_at > ? # --- where: user_id = ? ``` --- # Page 105: stdin **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/stdin.md --- # stdin --- title: stdin latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/stdin page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/stdin.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/stdin.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** Self-Managed Consumes data piped to stdin, chopping it into individual messages according to the specified scanner. #### Common ```yml inputs: label: "" stdin: scanner: lines: {} auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" stdin: scanner: lines: {} auto_replay_nacks: true ``` ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#scanner)`scanner` The [scanner](../../scanners/about/) by which the stream of bytes consumed will be broken out into individual messages. Scanners are useful for processing large sources of data without holding the entirety of it within memory. For example, the `csv` scanner allows you to process individual CSV rows without loading the entire CSV file in memory at once. Requires version 4.25.0 or later. **Type**: `scanner` **Default**: ```yaml lines: {} ``` --- # Page 106: subprocess **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/subprocess.md --- # subprocess --- title: subprocess latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/subprocess page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/subprocess.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/subprocess.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/subprocess/)[Output](/redpanda-connect/components/outputs/subprocess/)[Processor](/redpanda-connect/components/processors/subprocess/) **Available in:** Self-Managed Executes a command, runs it as a subprocess, and consumes messages from it over stdout. #### Common ```yml inputs: label: "" subprocess: name: "" # No default (required) args: [] codec: lines restart_on_exit: false ``` #### Advanced ```yml inputs: label: "" subprocess: name: "" # No default (required) args: [] codec: lines restart_on_exit: false max_buffer: 65536 ``` Messages are consumed according to a specified codec. The command is executed once and if it terminates the input also closes down gracefully. Alternatively, the field `restart_on_close` can be set to `true` in order to have Redpanda Connect re-execute the command each time it stops. The field `max_buffer` defines the maximum message size able to be read from the subprocess. This value should be set significantly above the real expected maximum message size. The execution environment of the subprocess is the same as the Redpanda Connect instance, including environment variables and the current working directory. ## [](#fields)Fields ### [](#args)`args[]` A list of arguments to provide the command. **Type**: `array` **Default**: `[]` ### [](#codec)`codec` The way in which messages should be consumed from the subprocess. **Type**: `string` **Default**: `lines` **Options**: `lines` ### [](#max_buffer)`max_buffer` The maximum expected size of an individual message. **Type**: `int` **Default**: `65536` ### [](#name)`name` The command to execute as a subprocess. **Type**: `string` ```yaml # Examples: name: cat # --- name: sed # --- name: awk ``` ### [](#restart_on_exit)`restart_on_exit` Whether the command should be re-executed each time the subprocess ends. **Type**: `bool` **Default**: `false` --- # Page 107: tigerbeetle_cdc **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/tigerbeetle_cdc.md --- # tigerbeetle\_cdc --- title: tigerbeetle_cdc latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/tigerbeetle_cdc page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/tigerbeetle_cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/tigerbeetle_cdc.adoc page-git-created-date: "2025-09-25" page-git-modified-date: "2025-10-06" --- **Available in:** Self-Managed Introduced in version 4.65.0 Enables integration with [TigerBeetle’s Change Data Capture](https://docs.tigerbeetle.com/operating/cdc/) (CDC) streaming for Redpanda Connect. This input component connects to a TigerBeetle cluster and continuously listens for real-time changes (such as account updates, transfers, and other ledger events). Each detected change is converted into a message and ingested into Redpanda Connect for further processing. Use this input when you need to synchronize TigerBeetle’s financial ledger events with downstream systems, enable event-driven architectures, or build real-time analytics pipelines. Typical use cases include auditing, monitoring, fraud detection, and maintaining up-to-date replicas of financial data in other services. ```yaml # Config fields, showing default values input: label: "" tigerbeetle_cdc: cluster_id: "" # Required: The unique 128-bit TigerBeetle cluster ID addresses: # Required: List of IP addresses for all TigerBeetle replicas - "" - "" - "" progress_cache: "" # Required: Cache resource name to track progress rate_limit: "" # Optional: Throttles request rate (defaults to no limit) event_count_max: 2730 # Optional: Max events per request (default 2730) idle_interval_ms: 1000 # Optional: Wait ms before querying again if no events (default 1000) timestamp_initial: "" # Optional: TigerBeetle timestamp to start from (nanosecond precision) auto_replay_nacks: true # Optional: Auto-replay failed batches (default true) ``` Each message produced by this input is a JSON object representing a TigerBeetle CDC event. The structure includes details about the event type, ledger, transfer, debit and credit accounts, and associated timestamps. For example: ```json { "timestamp": "1745328372758695656", "type": "single_phase", "ledger": 2, "transfer": { "id": "9082709", "amount": "3794", "pending_id": "0", "user_data_128": "79248595801719937611592367840129079151", "user_data_64": "13615171707598273871", "user_data_32": 3229992513, "timeout": 0, "code": 20295, "flags": 0, "timestamp": "1745328372758695656" }, "debit_account": { "id": "3750", "debits_pending": "0", "debits_posted": "8463768", "credits_pending": "0", "credits_posted": "8861179", "user_data_128": "118966247877720884212341541320399553321", "user_data_64": "526432537153007844", "user_data_32": 4157247332, "code": 1, "flags": 0, "timestamp": "1745328270103398016" }, "credit_account": { "id": "6765", "debits_pending": "0", "debits_posted": "8669204", "credits_pending": "0", "credits_posted": "8637251", "user_data_128": "43670023860556310170878798978091998141", "user_data_64": "12485093662256535374", "user_data_32": 1924162092, "code": 1, "flags": 0, "timestamp": "1745328270103401031" } } ``` ## [](#requirements)Requirements Before using the TigerBeetle CDC input, ensure that your environment meets the following requirements: - TigerBeetle cluster version 0.16.57 or later is required. - The TigerBeetle client version must not be newer than the cluster version to ensure compatibility. - This component is only available in cgo-enabled builds of Redpanda Connect (not in the Redpanda CLI or Docker image). You can either [download a prebuilt cgo-enabled binary](../../../install/prebuilt-binary/) or [build Redpanda Connect from source with cgo enabled](../../../install/build-from-source/). - Consumers must implement idempotency when processing messages to handle potential duplicates. ## [](#metadata)Metadata This input adds the following metadata fields to each message: | Metadata Field | Description | | --- | --- | | event_type | One of "single_phase", "two_phase_pending", "two_phase_posted", "two_phase_voided", or "two_phase_expired". | | ledger | The ledger code. | | transfer_code | The transfer code. | | debit_account_code | The debit account code. | | credit_account_code | The credit account code. | | timestamp | The unique event timestamp with nanosecond resolution. | | timestamp_ms | The event timestamp with millisecond resolution. | ## [](#guarantees)Guarantees This input guarantees at-least-once semantics, and makes a best effort to prevent duplicate messages. However, during crash recovery, it may replay unacknowledged messages that could have been already delivered to consumers. It is the consumer’s responsibility to perform idempotency checks when processing messages. ## [](#fields)Fields ### [](#cluster_id)`cluster_id` The TigerBeetle unique 128-bit cluster ID. **Type**: `string` ### [](#addresses)`addresses` A list of IP addresses of all the TigerBeetle replicas in the cluster. The order of addresses must correspond to the order of replicas. **Type**: `array` ### [](#progress_cache)`progress_cache` A [cache resource](../../caches/about/) used to track progress by storing the last acknowledged timestamp. This allows Redpanda Connect to resume from the latest delivered event upon restart. **Type**: `string` ### [](#rate_limit)`rate_limit` An optional [rate limit](../../rate_limits/about/) to throttle the number of **requests** made to TigerBeetle. **Type**: `string` **Default**: `""` ### [](#event_count_max)`event_count_max` The maximum number of events fetched from TigerBeetle per **request**. Must be greater than zero. **Type**: `int` **Default**: `2730` ### [](#idle_interval_ms)`idle_interval_ms` The time interval in milliseconds to wait before querying again when the last request returned no events. Must be greater than zero. **Type**: `int` **Default**: `1000` ### [](#timestamp_initial)`timestamp_initial` The initial timestamp to start extracting events from. If not defined, all CDC events available in the TigerBeetle cluster will be included. Ignored if a more recent timestamp has already been acknowledged. This is a TigerBeetle timestamp with nanosecond precision. **Type**: `string` **Default**: `""` ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level are automatically replayed indefinitely. If the cause of rejection persists, this can result in backpressure. If set to `false`, these messages are deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` --- # Page 108: timeplus **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/timeplus.md --- # timeplus --- title: timeplus latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/timeplus page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/timeplus.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/timeplus.adoc page-git-created-date: "2024-11-19" page-git-modified-date: "2025-10-06" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/timeplus/)[Output](/redpanda-connect/components/outputs/timeplus/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/inputs/timeplus/ "View the Cloud version of this component"), Self-Managed Executes a streaming or table query on [Timeplus Enterprise (Cloud or Self-Hosted)](https://docs.timeplus.com/) or the `timeplusd` component, and creates a structured message for each table row received. If you execute a streaming query, this input runs until the query terminates. For table queries, it shuts down after all rows returned by the query are exhausted. ```yml inputs: label: "" timeplus: query: "" # No default (required) url: tcp://localhost:8463 workspace: "" # No default (optional) apikey: "" # No default (optional) username: "" # No default (optional) password: "" # No default (optional) ``` ## [](#examples)Examples ### [](#from-timeplus-enterprise-cloud-via-http)From Timeplus Enterprise Cloud via HTTP You will need to create API Key on Timeplus Enterprise Cloud Web console first and then set the `apikey` field. ```yaml input: timeplus: url: https://us-west-2.timeplus.cloud workspace: my_workspace_id query: select * from iot apikey: ``` ### [](#from-timeplus-enterprise-self-hosted-via-http)From Timeplus Enterprise (self-hosted) via HTTP For self-hosted Timeplus Enterprise, you will need to specify the username and password as well as the URL of the App server ```yaml input: timeplus: url: http://localhost:8000 workspace: my_workspace_id query: select * from iot username: username password: pw ``` ### [](#from-timeplus-enterprise-self-hosted-via-tcp)From Timeplus Enterprise (self-hosted) via TCP Make sure the the schema of url is tcp ```yaml input: timeplus: url: tcp://localhost:8463 query: select * from iot username: timeplus password: timeplus ``` ## [](#fields)Fields ### [](#apikey)`apikey` The API key for the Timeplus Enterprise REST API. You need to generate the key in the web console of Timeplus Enterprise (Cloud). This field is required if you are reading messages from Timeplus Enterprise (Cloud). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#password)`password` The password for the Timeplus application server. This field is required if you are reading messages from Timeplus Enterprise (Self-Hosted) or `timeplusd`. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#query)`query` The query to execute on Timeplus Enterprise (Cloud or Self-Hosted) or `timeplusd`. **Type**: `string` ```yaml # Examples: query: select * from iot # --- query: select count(*) from table(iot) ``` ### [](#url)`url` The URL of your Timeplus instance, which should always include the schema and host. **Type**: `string` **Default**: `tcp://localhost:8463` ### [](#username)`username` The username for the Timeplus application server. This field is required if you are reading messages from Timeplus Enterprise (Self-Hosted) or `timeplusd`. **Type**: `string` ### [](#workspace)`workspace` The ID of the workspace you want to read messages from. This field is required if you are connecting to Timeplus Enterprise (Cloud or Self-Hosted) using HTTP. **Type**: `string` --- # Page 109: twitter_search **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/twitter_search.md --- # twitter\_search --- title: twitter_search latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/twitter_search page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/twitter_search.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/twitter_search.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Available in:** Self-Managed Consumes tweets matching a given search using the Twitter recent search V2 API. #### Common ```yml inputs: label: "" twitter_search: query: "" # No default (required) tweet_fields: [] poll_period: 1m backfill_period: 5m cache: "" # No default (required) api_key: "" # No default (required) api_secret: "" # No default (required) ``` #### Advanced ```yml inputs: label: "" twitter_search: query: "" # No default (required) tweet_fields: [] poll_period: 1m backfill_period: 5m cache: "" # No default (required) cache_key: last_tweet_id rate_limit: "" api_key: "" # No default (required) api_secret: "" # No default (required) ``` Continuously polls the [Twitter recent search V2 API](https://developer.twitter.com/en/docs/twitter-api/tweets/search/api-reference/get-tweets-search-recent) for tweets that match a given search query. Each tweet received is emitted as a JSON object message, with a field `id` and `text` by default. Extra fields [can be obtained from the search API](https://developer.twitter.com/en/docs/twitter-api/fields) when listed with the `tweet_fields` field. In order to paginate requests that are made the ID of the latest received tweet is stored in a [cache resource](../../caches/about/), which is then used by subsequent requests to ensure only tweets after it are consumed. It is recommended that the cache you use is persistent so that Redpanda Connect can resume searches at the correct place on a restart. Authentication is done using OAuth 2.0 credentials which can be generated within the [Twitter developer portal](https://developer.twitter.com). ## [](#fields)Fields ### [](#api_key)`api_key` An API key for OAuth 2.0 authentication. It is recommended that you populate this field using [environment variables](../../../configuration/interpolation/). **Type**: `string` ### [](#api_secret)`api_secret` An API secret for OAuth 2.0 authentication. It is recommended that you populate this field using [environment variables](../../../configuration/interpolation/). **Type**: `string` ### [](#backfill_period)`backfill_period` A duration string indicating the maximum age of tweets to acquire when starting a search. **Type**: `string` **Default**: `5m` ### [](#cache)`cache` A cache resource to use for request pagination. **Type**: `string` ### [](#cache_key)`cache_key` The key identifier used when storing the ID of the last tweet received. **Type**: `string` **Default**: `last_tweet_id` ### [](#poll_period)`poll_period` The length of time (as a duration string) to wait between each search request. This field can be set empty, in which case requests are made at the limit set by the rate limit. This field also supports cron expressions. **Type**: `string` **Default**: `1m` ### [](#query)`query` A search expression to use. **Type**: `string` ### [](#rate_limit)`rate_limit` An optional rate limit resource to restrict API requests with. **Type**: `string` **Default**: `""` ### [](#tweet_fields)`tweet_fields[]` An optional list of additional fields to obtain for each tweet, by default only the fields `id` and `text` are returned. For more info refer to the [twitter API docs](https://developer.twitter.com/en/docs/twitter-api/fields). **Type**: `array` **Default**: `[]` --- # Page 110: websocket **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/websocket.md --- # websocket --- title: websocket latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/websocket page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/websocket.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/websocket.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/websocket/)[Output](/redpanda-connect/components/outputs/websocket/) **Available in:** Self-Managed Connects to a websocket server and continuously receives messages. #### Common ```yml inputs: label: "" websocket: url: "" # No default (required) auto_replay_nacks: true ``` #### Advanced ```yml inputs: label: "" websocket: url: "" # No default (required) proxy_url: "" # No default (optional) open_message: "" # No default (optional) open_message_type: binary auto_replay_nacks: true tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] connection: max_retries: "" # No default (optional) oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} ``` It is possible to configure an `open_message`, which when set to a non-empty string will be sent to the websocket server each time a connection is first established. ## [](#fields)Fields ### [](#auto_replay_nacks)`auto_replay_nacks` Whether messages that are rejected (nacked) at the output level should be automatically replayed indefinitely, eventually resulting in back pressure if the cause of the rejections is persistent. If set to `false` these messages will instead be deleted. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data can be discarded immediately upon consumption and mutation. **Type**: `bool` **Default**: `true` ### [](#basic_auth)`basic_auth` Allows you to specify basic authentication. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#connection)`connection` Customise how websocket connection attempts are made. **Type**: `object` ### [](#connection-max_retries)`connection.max_retries` An optional limit to the number of consecutive retry attempts that will be made before abandoning the connection altogether and gracefully terminating the input. When all inputs terminate in this way the service (or stream) will shut down. If set to zero connections will never be reattempted upon a failure. If set below zero this field is ignored (effectively unset). **Type**: `int` ```yaml # Examples: max_retries: -1 # --- max_retries: 10 ``` ### [](#jwt)`jwt` Beta Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWT tokens provide secure, stateless authentication between services. **Type**: `object` ### [](#jwt-claims)`jwt.claims` A value used to identify the claims that issued the JWT. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` The cryptographic algorithm used to sign the JWT token. Supported algorithms include RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#oauth)`oauth` Configure OAuth version 1.0 authentication for secure API access. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` A value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` A secret provided in order to establish ownership of a given access token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` A value used to identify the client to the service provider. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` A secret used to establish ownership of the consumer key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#open_message)`open_message` An optional message to send to the server upon connection. **Type**: `string` ### [](#open_message_type)`open_message_type` An optional flag to indicate the data type of open\_message. **Type**: `string` **Default**: `binary` | Option | Summary | | --- | --- | | binary | Binary data open_message. | | text | Text data open_message. The text message payload is interpreted as UTF-8 encoded text data. | ### [](#proxy_url)`proxy_url` An optional HTTP proxy URL. **Type**: `string` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL to connect to. **Type**: `string` ```yaml # Examples: url: ws://localhost:4195/get/ws ``` --- # Page 111: zmq4 **URL**: https://docs.redpanda.com/redpanda-connect/components/inputs/zmq4.md --- # zmq4 --- title: zmq4 latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: inputs/zmq4 page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: inputs/zmq4.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/inputs/zmq4.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Input ▼ [Input](/redpanda-connect/components/inputs/zmq4/)[Output](/redpanda-connect/components/outputs/zmq4/) **Available in:** Self-Managed Consumes messages from a ZeroMQ socket. #### Common ```yml inputs: label: "" zmq4: urls: [] # No default (required) bind: false socket_type: "" # No default (required) sub_filters: [] ``` #### Advanced ```yml inputs: label: "" zmq4: urls: [] # No default (required) bind: false socket_type: "" # No default (required) sub_filters: [] high_water_mark: 0 poll_timeout: 5s ``` By default Redpanda Connect does not build with components that require linking to external libraries. If you wish to build Redpanda Connect locally with this component then set the build tag `x_benthos_extra`: ```bash # With go go install -tags "x_benthos_extra" github.com/redpanda-data/benthos/v4/cmd/benthos@latest # Using make make TAGS=x_benthos_extra ``` There is a specific docker tag postfix `-cgo` for C builds containing this component. ## [](#fields)Fields ### [](#urls)`urls` A list of URLs to connect to. If an item of the list contains commas it will be expanded into multiple URLs. **Type**: `array` ```yml # Examples urls: - tcp://localhost:5555 ``` ### [](#bind)`bind` Whether to bind to the specified URLs (otherwise they are connected to). **Type**: `bool` **Default**: `false` ### [](#socket_type)`socket_type` The socket type to connect as. **Type**: `string` Options: `PULL` , `SUB` . ### [](#sub_filters)`sub_filters` A list of subscription topic filters to use when consuming from a SUB socket. Specifying a single sub\_filter of `''` will subscribe to everything. **Type**: `array` **Default**: `[]` ### [](#high_water_mark)`high_water_mark` The message high water mark to use. **Type**: `int` **Default**: `0` ### [](#poll_timeout)`poll_timeout` The poll timeout to use. **Type**: `string` **Default**: `"5s"` --- # Page 112: Logger **URL**: https://docs.redpanda.com/redpanda-connect/components/logger/about.md --- # Logger --- title: Logger latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: logger/about page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: logger/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/logger/about.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-10-04" --- Redpanda Connect logging prints to stdout (or stderr if your output is stdout) and is formatted as [logfmt](https://brandur.org/logfmt) by default. Use these configuration options to change both the logging formats as well as the destination of logs. > 💡 **TIP** > > You can also configure the [`redpanda` component](../../redpanda/about/) to send logs and status updates to topics on a Redpanda cluster. #### Common ```yaml # Common config fields, showing default values logger: level: INFO format: logfmt add_timestamp: false static_fields: '@service': redpanda-connect ``` #### Advanced ```yaml # All config fields, showing default values logger: level: INFO format: logfmt add_timestamp: false level_name: level timestamp_name: time message_name: msg static_fields: '@service': redpanda-connect file: path: "" rotate: false rotate_max_age_days: 0 ``` ## [](#fields)Fields The schema of the `logger` section is as follows: ### [](#level)`level` Set the minimum severity level for emitting logs. **Type**: `string` **Default**: `"INFO"` Options: `OFF` , `FATAL` , `ERROR` , `WARN` , `INFO` , `DEBUG` , `TRACE` , `ALL` , `NONE` ### [](#format)`format` Set the format of emitted logs. **Type**: `string` **Default**: `"logfmt"` Options: `json` , `logfmt` ### [](#add_timestamp)`add_timestamp` Whether to include timestamps in logs. **Type**: `bool` **Default**: `false` ### [](#level_name)`level_name` The name of the level field added to logs when the `format` is `json`. **Type**: `string` **Default**: `"level"` ### [](#timestamp_name)`timestamp_name` The name of the timestamp field added to logs when `add_timestamp` is set to `true` and the `format` is `json`. **Type**: `string` **Default**: `"time"` ### [](#message_name)`message_name` The name of the message field added to logs when the `format` is `json`. **Type**: `string` **Default**: `"msg"` ### [](#static_fields)`static_fields` A map of key/value pairs to add to each structured log. **Type**: `object` **Default**: `{"@service":"redpanda-connect"}` ### [](#file)`file` Experimental: Specify fields for optionally writing logs to a file. **Type**: `object` ### [](#file-path)`file.path` The file path to write logs to, if the file does not exist it will be created. Leave this field empty or unset to disable file based logging. **Type**: `string` **Default**: `""` ### [](#file-rotate)`file.rotate` Whether to rotate log files automatically. **Type**: `bool` **Default**: `false` ### [](#file-rotate_max_age_days)`file.rotate_max_age_days` The maximum number of days to retain old log files based on the timestamp encoded in their filename, after which they are deleted. Setting to zero disables this mechanism. **Type**: `int` **Default**: `0` --- # Page 113: Metrics **URL**: https://docs.redpanda.com/redpanda-connect/components/metrics/about.md --- # Metrics --- title: Metrics latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: metrics/about page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: metrics/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/metrics/about.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-09" --- Redpanda Connect emits lots of metrics in order to expose how components configured within your pipeline are behaving. You can configure exactly where these metrics end up with the config field `metrics`, which describes a metrics format and destination. For example, if you wished to push them via the StatsD protocol you could use this configuration: ```yaml metrics: statsd: address: localhost:8125 flush_period: 100ms ``` The default metrics configuration is to expose Prometheus metrics on the [service-wide HTTP endpoint](../../http/about/) at the endpoints `/metrics` and `/stats`. ## [](#timings)Timings It’s worth noting that timing metrics within Redpanda Connect are measured in nanoseconds and are therefore named with a `_ns` suffix. However, some exporters do not support this level of precision and are downgraded, or have the unit converted for convenience. In these cases the exporter documentation outlines the conversion and why it is made. ## [](#metric-names)Metric names Each major Redpanda Connect component type emits one or more metrics with the name prefixed by the type. These metrics are intended to provide an overview of behavior, performance and health. Some specific component implementations may provide their own unique metrics on top of these standardized ones, these extra metrics can be found listed on their respective documentation pages. ## [](#inputs)Inputs - `input_received`: A count of the number of messages received by the input. - `input_latency_ns`: Measures the roundtrip latency in nanoseconds from the point at which a message is read up to the moment the message has either been acknowledged by an output, has been stored within a buffer, or has been rejected (nacked). - `batch_created`: A count of each time an input-level batch has been created using a batching policy. Includes a label `mechanism` describing the particular mechanism that triggered it, one of; `count`, `size`, `period`, `check`. - `input_connection_up`: For continuous stream based inputs represents a count of the number of the times the input has successfully established a connection to the target source. For poll based inputs that do not retain an active connection this value will increment once. - `input_connection_failed`: For continuous stream based inputs represents a count of the number of times the input has failed to establish a connection to the target source. - `input_connection_lost`: For continuous stream based inputs represents a count of the number of times the input has lost a previously established connection to the target source. > ⚠️ **CAUTION** > > The behavior of connection metrics may differ based on input type due to certain libraries and protocols obfuscating the concept of a single connection. ### [](#buffers)Buffers - `buffer_received`: A count of the number of messages written to the buffer. - `buffer_batch_received`: A count of the number of message batches written to the buffer. - `buffer_sent`: A count of the number of messages read from the buffer. - `buffer_batch_sent`: A count of the number of message batches read from the buffer. - `buffer_latency_ns`: Measures the roundtrip latency in nanoseconds from the point at which a message is read from the buffer up to the moment it has been acknowledged by the output. - `batch_created`: A count of each time a buffer-level batch has been created using a batching policy. Includes a label `mechanism` describing the particular mechanism that triggered it, one of; `count`, `size`, `period`, `check`. ### [](#processors)Processors - `processor_received`: A count of the number of messages the processor has been executed upon. - `processor_batch_received`: A count of the number of message batches the processor has been executed upon. - `processor_sent`: A count of the number of messages the processor has returned. - `processor_batch_sent`: A count of the number of message batches the processor has returned. - `processor_error`: A count of the number of times the processor has errored. In cases where an error is batch-wide the count is incremented by one, and therefore would not match the number of messages. - `processor_latency_ns`: Latency of message processing in nanoseconds. When a processor acts upon a batch of messages this latency measures the time taken to process all messages of the batch. ### [](#outputs)Outputs - `output_sent`: A count of the number of messages sent by the output. - `output_batch_sent`: A count of the number of message batches sent by the output. - `output_error`: A count of the number of send attempts that have failed. On failed batched sends this count is incremented once only. - `output_latency_ns`: Latency of writes in nanoseconds. This metric may not be populated by outputs that are pull-based such as the `http_server`. - `batch_created`: A count of each time an output-level batch has been created using a batching policy. Includes a label `mechanism` describing the particular mechanism that triggered it, one of; `count`, `size`, `period`, `check`. - `output_connection_up`: For continuous stream based outputs represents a count of the number of the times the output has successfully established a connection to the target sink. For poll based outputs that do not retain an active connection this value will increment once. - `output_connection_failed`: For continuous stream based outputs represents a count of the number of times the output has failed to establish a connection to the target sink. - `output_connection_lost`: For continuous stream based outputs represents a count of the number of times the output has lost a previously established connection to the target sink. > ⚠️ **CAUTION** > > The behavior of connection metrics may differ based on output type due to certain libraries and protocols obfuscating the concept of a single connection. ### [](#caches)Caches All cache metrics have a label `operation` denoting the operation that triggered the metric series, one of; `add`, `get`, `set` or `delete`. - `cache_success`: A count of the number of successful cache operations. - `cache_error`: A count of the number of cache operations that resulted in an error. - `cache_latency_ns`: Latency of operations in nanoseconds. - `cache_not_found`: A count of the number of get operations that yielded no value due to the item not being found. This count is separate from `cache_error`. - `cache_duplicate`: A count of the number of add operations that were aborted due to the key already existing. This count is separate from `cache_error`. ### [](#rate-limits)Rate limits - `rate_limit_checked`: A count of the number of times the rate limit has been probed. - `rate_limit_triggered`: A count of the number of times the rate limit has been triggered by a probe. - `rate_limit_error`: A count of the number of times the rate limit has errored when probed. ## [](#metric-labels)Metric labels The standard metric names are unique to the component type, but a benthos config may consist of any number of component instantiations. In order to provide a metrics series that is unique for each instantiation Redpanda Connect adds labels (or tags) that uniquely identify the instantiation. These labels are as follows: ### [](#path)`path` The `path` label contains a string representation of the position of a component instantiation within a config in a format that would locate it within a Bloblang mapping, beginning at `root`. This path is a best attempt and may not exactly represent the source component position in all cases and is intended to be used for assisting observability only. This is the highest cardinality label since paths will change as configs are updated and expanded. It is therefore worth removing this label with a [mapping](#metric-mapping) in cases where you wish to restrict the number of unique metric series. ### [](#label)`label` The `label` label contains the unique label configured for a component emitting the metric series, or is empty for components that do not have a configured label. This is the most useful label for uniquely identifying a series for a component. ### [](#stream)`stream` The `stream` label is present in a metric series emitted from a stream config executed when Redpanda Connect is running in [streams mode](../../../guides/streams_mode/about/), and is populated with the stream name. ## [](#example)Example The following Redpanda Connect configuration: ```yaml input: label: foo http_server: {} pipeline: processors: - mapping: | root.message = this root.meta.link_count = this.links.length() root.user.age = this.user.age.number() output: label: bar stdout: {} metrics: prometheus: {} ``` Would produce the following metrics series: ```text input_latency_ns{label="foo",path="root.input"} input_received{endpoint="post",label="foo",path="root.input"} input_received{endpoint="websocket",label="foo",path="root.input"} processor_batch_received{label="",path="root.pipeline.processors.0"} processor_batch_sent{label="",path="root.pipeline.processors.0"} processor_error{label="",path="root.pipeline.processors.0"} processor_latency_ns{label="",path="root.pipeline.processors.0"} processor_received{label="",path="root.pipeline.processors.0"} processor_sent{label="",path="root.pipeline.processors.0"} output_batch_sent{label="bar",path="root.output"} output_connection_failed{label="bar",path="root.output"} output_connection_lost{label="bar",path="root.output"} output_connection_up{label="bar",path="root.output"} output_error{label="bar",path="root.output"} output_latency_ns{label="bar",path="root.output"} output_sent{label="bar",path="root.output"} ``` ## [](#metric-mapping)Metric mapping Since Redpanda Connect emits a large variety of metrics it is often useful to restrict or modify the metrics that are emitted. This can be done using the [Bloblang mapping language](../../../guides/bloblang/about/) in the field `metrics.mapping`. This is a mapping executed for each metric that is registered within the Redpanda Connect service and allows you to delete an entire series, modify the series name and delete or modify individual labels. Within the mapping the input document (referenced by the keyword `this`) is a string value containing the metric name, and the resulting document (referenced by the keyword `root`) must be a string value containing the resulting name. As is standard in Bloblang mappings, if the value of `root` is not assigned within the mapping then the metric name remains unchanged. If the value of `root` is `deleted()` then the metric series is dropped. Labels can be referenced as metadata values with the function `meta`, where if the label does not exist in the series being mapped the value `null` is returned. Labels can be changed by using meta assignments, and can be assigned `deleted()` in order to remove them. For example, the following mapping removes all but the `label` label entirely, which reduces the cardinality of each series. It also renames the `label` (for some reason) so that labels containing meows now contain woofs. Finally, the mapping restricts the metrics emitted to only three series; one for the input count, one for processor errors, and one for the output count, it does this by looking up metric names in a static array of allowed names, and if not present the `root` is assigned `deleted()`: ```yaml metrics: mapping: | # Delete all pre-existing labels meta = deleted() # Re-add the `label` label with meows replaced with woofs meta label = meta("label").replace("meow", "woof") # Delete all metric series that aren't in our list root = if ![ "input_received", "processor_error", "output_sent", ].contains(this) { deleted() } prometheus: use_histogram_timing: false ``` --- # Page 114: aws_cloudwatch **URL**: https://docs.redpanda.com/redpanda-connect/components/metrics/aws_cloudwatch.md --- # aws\_cloudwatch --- title: aws_cloudwatch latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: metrics/aws_cloudwatch page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: metrics/aws_cloudwatch.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/metrics/aws_cloudwatch.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** Self-Managed Send metrics to AWS CloudWatch using the PutMetricData endpoint. Introduced in version 3.36.0. #### Common ```yml metrics: aws_cloudwatch: namespace: Benthos ``` #### Advanced ```yml metrics: aws_cloudwatch: namespace: Benthos flush_period: 100ms region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` ## [](#timing-metrics)Timing metrics The smallest timing unit that CloudWatch supports is microseconds, therefore timing metrics are automatically downgraded to microseconds (by dividing delta values by 1000). This conversion will also apply to custom timing metrics produced with a `metric` processor. ## [](#billing)Billing AWS bills per metric series exported, it is therefore STRONGLY recommended that you reduce the metrics that are exposed with a `mapping` like this: ```yaml metrics: mapping: | if ![ "input_received", "input_latency", "output_sent", ].contains(this) { deleted() } aws_cloudwatch: namespace: Foo ``` ## [](#fields)Fields ### [](#credentials)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](../../../guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#flush_period)`flush_period` The period of time between PutMetricData requests. **Type**: `string` **Default**: `100ms` ### [](#namespace)`namespace` The namespace used to distinguish metrics from other services. **Type**: `string` **Default**: `Benthos` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#tcp)`tcp` TCP socket configuration. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` --- # Page 115: influxdb **URL**: https://docs.redpanda.com/redpanda-connect/components/metrics/influxdb.md --- # influxdb --- title: influxdb latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: metrics/influxdb page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: metrics/influxdb.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/metrics/influxdb.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** Self-Managed Send metrics to InfluxDB 1.x using the `/write` endpoint. Introduced in version 3.36.0. #### Common ```yml metrics: influxdb: url: "" # No default (required) db: "" # No default (required) ``` #### Advanced ```yml metrics: influxdb: url: "" # No default (required) db: "" # No default (required) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] username: "" password: "" include: runtime: "" debug_gc: "" interval: 1m ping_interval: 20s precision: s timeout: 5s tags: {} retention_policy: "" # No default (optional) write_consistency: "" # No default (optional) ``` See [https://docs.influxdata.com/influxdb/v1.8/tools/api/#write-http-endpoint](https://docs.influxdata.com/influxdb/v1.8/tools/api/#write-http-endpoint) for further details on the write API. ## [](#fields)Fields ### [](#db)`db` The name of the database to use. **Type**: `string` ### [](#include)`include` Optional additional metrics to collect, enabling these metrics may have some performance implications as it acquires a global semaphore and does `stoptheworld()`. **Type**: `object` ### [](#include-debug_gc)`include.debug_gc` A duration string indicating how often to poll and collect GC metrics. Leave empty to disable this metric. **Type**: `string` **Default**: `""` ```yaml # Examples: debug_gc: 1m ``` ### [](#include-runtime)`include.runtime` A duration string indicating how often to poll and collect runtime metrics. Leave empty to disable this metric **Type**: `string` **Default**: `""` ```yaml # Examples: runtime: 1m ``` ### [](#interval)`interval` A duration string indicating how often metrics should be flushed. **Type**: `string` **Default**: `1m` ### [](#password)`password` A password (when applicable). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#ping_interval)`ping_interval` A duration string indicating how often to ping the host. **Type**: `string` **Default**: `20s` ### [](#precision)`precision` \[ns|us|ms|s\] timestamp precision passed to write api. **Type**: `string` **Default**: `s` ### [](#retention_policy)`retention_policy` Sets the retention policy for each write. **Type**: `string` ### [](#tags)`tags` Global tags added to each metric. **Type**: `string` **Default**: `{}` ```yaml # Examples: tags: hostname: localhost zone: danger ``` ### [](#timeout)`timeout` How long to wait for response for both ping and writing metrics. **Type**: `string` **Default**: `5s` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` A URL of the format `[https|http|udp]://host:port` to the InfluxDB host. **Type**: `string` ### [](#username)`username` A username (when applicable). **Type**: `string` **Default**: `""` ### [](#write_consistency)`write_consistency` \[any|one|quorum|all\] sets write consistency when available. **Type**: `string` --- # Page 116: json_api **URL**: https://docs.redpanda.com/redpanda-connect/components/metrics/json_api.md --- # json\_api --- title: json_api latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: metrics/json_api page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: metrics/json_api.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/metrics/json_api.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** Self-Managed Serves metrics as JSON object with the service wide HTTP service at the endpoints `/stats` and `/metrics`. ```yml metrics: json_api: host: "" port: 4040 path: /stats ``` This metrics type is useful for debugging as it provides a human readable format that you can parse with tools such as `jq` --- # Page 117: logger **URL**: https://docs.redpanda.com/redpanda-connect/components/metrics/logger.md --- # logger --- title: logger latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: metrics/logger page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: metrics/logger.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/metrics/logger.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** Self-Managed Prints aggregated metrics through the logger. ```yml metrics: logger: push_interval: "" # No default (optional) flush_metrics: false ``` Prints each metric produced by Redpanda Connect as a log event (level `info` by default) during shutdown, and optionally on an interval. This metrics type is useful for debugging pipelines when you only have access to the logger output and not the service-wide server. Otherwise it’s recommended that you use either the `prometheus` or `json_api` types. ## [](#fields)Fields ### [](#flush_metrics)`flush_metrics` Whether counters and timing metrics should be reset to 0 each time metrics are printed. **Type**: `bool` **Default**: `false` ### [](#push_interval)`push_interval` An optional period of time to continuously print all metrics. **Type**: `string` --- # Page 118: none **URL**: https://docs.redpanda.com/redpanda-connect/components/metrics/none.md --- # none --- title: none latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: metrics/none page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: metrics/none.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/metrics/none.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- **Type:** Buffer ▼ [Buffer](/redpanda-connect/components/buffers/none/)[Metric](/redpanda-connect/components/metrics/none/)[Tracer](/redpanda-connect/components/tracers/none/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/buffers/none/ "View the Cloud version of this component"), Self-Managed Disable metrics entirely. ```yml # Config fields, showing default values metrics: none: {} mapping: "" ``` --- # Page 119: prometheus **URL**: https://docs.redpanda.com/redpanda-connect/components/metrics/prometheus.md --- # prometheus --- title: prometheus latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: metrics/prometheus page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: metrics/prometheus.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/metrics/prometheus.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/metrics/prometheus/ "View the Cloud version of this component"), Self-Managed Host endpoints (`/metrics` and `/stats`) for Prometheus scraping. #### Common ```yml metrics: prometheus: ``` #### Advanced ```yml metrics: prometheus: use_histogram_timing: false histogram_buckets: [] summary_quantiles_objectives: - error: 0.05 quantile: 0.5 - error: 0.01 quantile: 0.9 - error: 0.001 quantile: 0.99 add_process_metrics: false add_go_metrics: false push_url: "" # No default (optional) push_interval: "" # No default (optional) push_job_name: benthos_push push_basic_auth: username: "" password: "" file_output_path: "" ``` ## [](#fields)Fields ### [](#add_go_metrics)`add_go_metrics` Whether to export Go runtime metrics such as GC pauses in addition to Redpanda Connect metrics. **Type**: `bool` **Default**: `false` ### [](#add_process_metrics)`add_process_metrics` Whether to export process metrics such as CPU and memory usage in addition to Redpanda Connect metrics. **Type**: `bool` **Default**: `false` ### [](#file_output_path)`file_output_path` An optional file path to write all prometheus metrics on service shutdown. **Type**: `string` **Default**: `""` ### [](#histogram_buckets)`histogram_buckets[]` Timing metrics histogram buckets (in seconds). If left empty defaults to DefBuckets ([https://pkg.go.dev/github.com/prometheus/client\_golang/prometheus#pkg-variables](https://pkg.go.dev/github.com/prometheus/client_golang/prometheus#pkg-variables)). Applicable when `use_histogram_timing` is set to `true`. Requires version 3.63.0 or later. **Type**: `float` **Default**: `[]` ### [](#push_basic_auth)`push_basic_auth` The Basic Authentication credentials. **Type**: `object` ### [](#push_basic_auth-password)`push_basic_auth.password` The Basic Authentication password. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#push_basic_auth-username)`push_basic_auth.username` The Basic Authentication username. **Type**: `string` **Default**: `""` ### [](#push_interval)`push_interval` The period of time between each push when sending metrics to a Push Gateway. **Type**: `string` ### [](#push_job_name)`push_job_name` An identifier for push jobs. **Type**: `string` **Default**: `benthos_push` ### [](#push_url)`push_url` An optional [Push Gateway URL](#push-gateway) to push metrics to. **Type**: `string` ### [](#summary_quantiles_objectives)`summary_quantiles_objectives[]` A list of timing metrics summary buckets (as quantiles). Applicable when `use_histogram_timing` is set to `false`. Requires version 4.23.0 or later. **Type**: `object` **Default**: ```yaml - error: 0.05 quantile: 0.5 - error: 0.01 quantile: 0.9 - error: 0.001 quantile: 0.99 ``` ```yaml # Examples: summary_quantiles_objectives: - error: 0.05 quantile: 0.5 - error: 0.01 quantile: 0.9 - error: 0.001 quantile: 0.99 ``` ### [](#summary_quantiles_objectives-error)`summary_quantiles_objectives[].error` Permissible margin of error for quantile calculations. Precise calculations in a streaming context (without prior knowledge of the full dataset) can be resource-intensive. To balance accuracy with computational efficiency, an error margin is introduced. For instance, if the 90th quantile (`0.9`) is determined to be `100ms` with a 1% error margin (`0.01`), the true value will fall within the `[99ms, 101ms]` range.) **Type**: `float` **Default**: `0` ### [](#summary_quantiles_objectives-quantile)`summary_quantiles_objectives[].quantile` Quantile value. **Type**: `float` **Default**: `0` ### [](#use_histogram_timing)`use_histogram_timing` Whether to export timing metrics as a histogram, if `false` a summary is used instead. When exporting histogram timings the delta values are converted from nanoseconds into seconds in order to better fit within bucket definitions. For more information on histograms and summaries refer to: [https://prometheus.io/docs/practices/histograms/](https://prometheus.io/docs/practices/histograms/). Requires version 3.63.0 or later. **Type**: `bool` **Default**: `false` ## [](#push-gateway)Push gateway The field `push_url` is optional and when set will trigger a push of metrics to a [Prometheus Push Gateway](https://prometheus.io/docs/instrumenting/pushing/) once Redpanda Connect shuts down. It is also possible to specify a `push_interval` which results in periodic pushes. The Push Gateway is useful for when Redpanda Connect instances are short lived. Do not include the "/metrics/jobs/…​" path in the push URL. If the Push Gateway requires HTTP Basic Authentication it can be configured with `push_basic_auth`. --- # Page 120: statsd **URL**: https://docs.redpanda.com/redpanda-connect/components/metrics/statsd.md --- # statsd --- title: statsd latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: metrics/statsd page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: metrics/statsd.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/metrics/statsd.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** Self-Managed Pushes metrics using the [StatsD protocol](https://github.com/statsd/statsd). Supported tagging formats are 'none', 'datadog' and 'influxdb'. ```yml metrics: statsd: address: "" # No default (required) flush_period: 100ms tag_format: none ``` ## [](#fields)Fields ### [](#address)`address` The address to send metrics to. **Type**: `string` ### [](#flush_period)`flush_period` The time interval between metrics flushes. **Type**: `string` **Default**: `100ms` ### [](#tag_format)`tag_format` Metrics tagging is supported in a variety of formats. **Type**: `string` **Default**: `none` **Options**: `none`, `datadog`, `influxdb` ### [](#tags)`tags` Global tags added to each metric. **Type**: `string` **Default**: `{}` ```yaml # Examples: tags: hostname: localhost zone: danger ``` --- # Page 121: Outputs **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/about.md --- # Outputs --- title: Outputs latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/about page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/about.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- An output is a sink where we wish to send our consumed data after applying an optional array of [processors](../../processors/about/). Only one output is configured at the root of a Redpanda Connect config. However, the output can be a [broker](../broker/) which combines multiple outputs under a chosen brokering pattern, or a [switch](../switch/) which is used to multiplex against different outputs. An output config section looks like this: ```yaml output: label: my_s3_output aws_s3: bucket: TODO path: '${! meta("kafka_topic") }/${! json("message.id") }.json' # Optional list of processing steps processors: - mapping: '{"message":this,"meta":{"link_count":this.links.length()}}' ``` ## [](#back-pressure)Back pressure Redpanda Connect outputs apply back pressure to components upstream. This means if your output target starts blocking traffic Redpanda Connect will gracefully stop consuming until the issue is resolved. ## [](#retries)Retries When a Redpanda Connect output fails to send a message the error is propagated back up to the input, where depending on the protocol it will either be pushed back to the source as a Noack (e.g. AMQP) or will be reattempted indefinitely with the commit withheld until success (e.g. Kafka). It’s possible to instead have Redpanda Connect indefinitely retry an output until success with a [`retry`](../retry/) output. Some other outputs, such as the [`broker`](../broker/), might also retry indefinitely depending on their configuration. ## [](#dead-letter-queues)Dead letter queues It’s possible to create fallback outputs for when an output target fails using a [`fallback`](../fallback/) output: ```yaml output: fallback: - aws_sqs: url: https://sqs.us-west-2.amazonaws.com/TODO/TODO max_in_flight: 20 - http_client: url: http://backup:1234/dlq verb: POST ``` ## [](#multiplexing-outputs)Multiplexing outputs There are a few different ways of multiplexing in Redpanda Connect, here’s a quick run through: ### [](#interpolation-multiplexing)Interpolation multiplexing Some output fields support [field interpolation](../../../configuration/interpolation/), which is a super easy way to multiplex messages based on their contents in situations where you are multiplexing to the same service. For example, multiplexing against Kafka topics is a common pattern: ```yaml output: kafka: addresses: [ TODO:6379 ] topic: ${! meta("target_topic") } ``` Refer to the field documentation for a given output to see if it support interpolation. ### [](#switch-multiplexing)Switch multiplexing A more advanced form of multiplexing is to route messages to different output configurations based on a query. This is easy with the [`switch` output](../switch/): ```yaml output: switch: cases: - check: this.type == "foo" output: amqp_1: urls: [ amqps://guest:guest@localhost:5672/ ] target_address: queue:/the_foos - check: this.type == "bar" output: gcp_pubsub: project: dealing_with_mike topic: mikes_bars - output: redis_streams: url: tcp://localhost:6379 stream: everything_else processors: - mapping: | root = this root.type = this.type.not_null() | "unknown" ``` ## [](#labels)Labels Outputs have an optional field `label` that can uniquely identify them in observability data such as metrics and logs. This can be useful when running configs with multiple outputs, otherwise their metrics labels will be generated based on their composition. For more information check out the [metrics documentation](../../metrics/about/). ## [](#categories)Categories ### Services Outputs that write to storage or message streaming services. [**RabbitMQ**](/redpanda-connect/components/outputs/amqp_0_9/) [**RabbitMQ**](/redpanda-connect/components/outputs/amqp_1/) [**AWS DynamoDB**](/redpanda-connect/components/outputs/aws_dynamodb/) [**AWS Kinesis**](/redpanda-connect/components/outputs/aws_kinesis/) [**AWS Kinesis Firehose**](/redpanda-connect/components/outputs/aws_kinesis_firehose/) [**AWS S3**](/redpanda-connect/components/outputs/aws_s3/) [**AWS SNS**](/redpanda-connect/components/outputs/aws_sns/) [**AWS SQS**](/redpanda-connect/components/outputs/aws_sqs/) [**Azure Blob Storage**](/redpanda-connect/components/outputs/azure_blob_storage/) [**Microsoft Azure**](/redpanda-connect/components/outputs/azure_data_lake_gen2/) [**Azure Queue Storage**](/redpanda-connect/components/outputs/azure_queue_storage/) [**Azure Table Storage**](/redpanda-connect/components/outputs/azure_table_storage/) [**beanstalkd**](/redpanda-connect/components/outputs/beanstalkd/) [**cache**](/redpanda-connect/components/outputs/cache/) [**cypher**](/redpanda-connect/components/outputs/cypher/) [**discord**](/redpanda-connect/components/outputs/discord/) [**elasticsearch\_v8**](/redpanda-connect/components/outputs/elasticsearch_v8/) [**elasticsearch\_v9**](/redpanda-connect/components/outputs/elasticsearch_v9/) [**GCP BigQuery**](/redpanda-connect/components/outputs/gcp_bigquery/) [**GCP Cloud Storage**](/redpanda-connect/components/outputs/gcp_cloud_storage/) [**GCP PubSub**](/redpanda-connect/components/outputs/gcp_pubsub/) [**hdfs**](/redpanda-connect/components/outputs/hdfs/) [**Apache Iceberg**](/redpanda-connect/components/outputs/iceberg/) [**MongoDB**](/redpanda-connect/components/outputs/mongodb/) [**mqtt**](/redpanda-connect/components/outputs/mqtt/) [**NATS**](/redpanda-connect/components/outputs/nats/) [**NATS JetStream**](/redpanda-connect/components/outputs/nats_jetstream/) [**NATS KV**](/redpanda-connect/components/outputs/nats_kv/) [**NATS Stream**](/redpanda-connect/components/outputs/nats_stream/) [**nsq**](/redpanda-connect/components/outputs/nsq/) [**ockam\_kafka**](/redpanda-connect/components/outputs/ockam_kafka/) [**opensearch**](/redpanda-connect/components/outputs/opensearch/) [**OpenTelemetry**](/redpanda-connect/components/outputs/otlp_grpc/) [**OpenTelemetry**](/redpanda-connect/components/outputs/otlp_http/) [**Apache Pulsar**](/redpanda-connect/components/outputs/pulsar/) [**pusher**](/redpanda-connect/components/outputs/pusher/) [**questdb**](/redpanda-connect/components/outputs/questdb/) [**Redis Hash**](/redpanda-connect/components/outputs/redis_hash/) [**Redis List**](/redpanda-connect/components/outputs/redis_list/) [**Redis PubSub**](/redpanda-connect/components/outputs/redis_pubsub/) [**Redis Streams**](/redpanda-connect/components/outputs/redis_streams/) [**redpanda**](/redpanda-connect/components/outputs/redpanda/) [**redpanda\_common**](/redpanda-connect/components/outputs/redpanda_common/) [**redpanda\_migrator**](/redpanda-connect/components/outputs/redpanda_migrator/) [**Salesforce**](/redpanda-connect/components/outputs/salesforce_sink/) [**Snowflake**](/redpanda-connect/components/outputs/snowflake_put/) [**Snowflake Streaming**](/redpanda-connect/components/outputs/snowflake_streaming/) [**Splunk**](/redpanda-connect/components/outputs/splunk_hec/) [**SQL**](/redpanda-connect/components/outputs/sql_insert/) [**SQL**](/redpanda-connect/components/outputs/sql_raw/) [**timeplus**](/redpanda-connect/components/outputs/timeplus/) ### AWS Outputs that write to Amazon Web Services products. [**AWS DynamoDB**](/redpanda-connect/components/outputs/aws_dynamodb/) [**AWS Kinesis**](/redpanda-connect/components/outputs/aws_kinesis/) [**AWS Kinesis Firehose**](/redpanda-connect/components/outputs/aws_kinesis_firehose/) [**AWS S3**](/redpanda-connect/components/outputs/aws_s3/) [**AWS SNS**](/redpanda-connect/components/outputs/aws_sns/) [**AWS SQS**](/redpanda-connect/components/outputs/aws_sqs/) [**Apache Iceberg**](/redpanda-connect/components/outputs/iceberg/) ### Azure Outputs that write to Microsoft Azure services. [**Azure Blob Storage**](/redpanda-connect/components/outputs/azure_blob_storage/) [**Microsoft Azure**](/redpanda-connect/components/outputs/azure_cosmosdb/) [**Microsoft Azure**](/redpanda-connect/components/outputs/azure_data_lake_gen2/) [**Azure Queue Storage**](/redpanda-connect/components/outputs/azure_queue_storage/) [**Azure Table Storage**](/redpanda-connect/components/outputs/azure_table_storage/) [**Apache Iceberg**](/redpanda-connect/components/outputs/iceberg/) ### Utility Outputs that provide utility by combining/wrapping other outputs. [**broker**](/redpanda-connect/components/outputs/broker/) [**drop**](/redpanda-connect/components/outputs/drop/) [**drop\_on**](/redpanda-connect/components/outputs/drop_on/) [**dynamic**](/redpanda-connect/components/outputs/dynamic/) [**fallback**](/redpanda-connect/components/outputs/fallback/) [**inproc**](/redpanda-connect/components/outputs/inproc/) [**reject**](/redpanda-connect/components/outputs/reject/) [**reject\_errored**](/redpanda-connect/components/outputs/reject_errored/) [**resource**](/redpanda-connect/components/outputs/resource/) [**retry**](/redpanda-connect/components/outputs/retry/) [**subprocess**](/redpanda-connect/components/outputs/subprocess/) [**switch**](/redpanda-connect/components/outputs/switch/) [**sync\_response**](/redpanda-connect/components/outputs/sync_response/) ### Integration [**couchbase**](/redpanda-connect/components/outputs/couchbase/) [**schema\_registry**](/redpanda-connect/components/outputs/schema_registry/) ### AI [**cyborgdb**](/redpanda-connect/components/outputs/cyborgdb/) [**pinecone**](/redpanda-connect/components/outputs/pinecone/) [**qdrant**](/redpanda-connect/components/outputs/qdrant/) ### Social Outputs that write to social applications and services. [**discord**](/redpanda-connect/components/outputs/discord/) ### Local Outputs that write to the local machine/filesystem. [**file**](/redpanda-connect/components/outputs/file/) [**stdout**](/redpanda-connect/components/outputs/stdout/) ### GCP Outputs that write to Google Cloud Platform services. [**GCP BigQuery**](/redpanda-connect/components/outputs/gcp_bigquery/) [**GCP Cloud Storage**](/redpanda-connect/components/outputs/gcp_cloud_storage/) [**GCP PubSub**](/redpanda-connect/components/outputs/gcp_pubsub/) [**Apache Iceberg**](/redpanda-connect/components/outputs/iceberg/) ### Network Outputs that write directly to low level network protocols. [**HTTP**](/redpanda-connect/components/outputs/http_client/) [**HTTP**](/redpanda-connect/components/outputs/http_server/) [**nanomsg**](/redpanda-connect/components/outputs/nanomsg/) [**sftp**](/redpanda-connect/components/outputs/sftp/) [**socket**](/redpanda-connect/components/outputs/socket/) [**websocket**](/redpanda-connect/components/outputs/websocket/) [**zmq4**](/redpanda-connect/components/outputs/zmq4/) ### [**Slack Reaction**](/redpanda-connect/components/outputs/slack_reaction/) --- # Page 122: amqp_0_9 **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/amqp_0_9.md --- # amqp\_0\_9 --- title: amqp_0_9 latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/amqp_0_9 page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/amqp_0_9.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/amqp_0_9.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/amqp_0_9/)[Input](/redpanda-connect/components/inputs/amqp_0_9/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/amqp_0_9/ "View the Cloud version of this component"), Self-Managed Sends messages to an AMQP (0.91) exchange. AMQP is a messaging protocol used by various message brokers, including RabbitMQ. #### Common ```yml outputs: label: "" amqp_0_9: urls: [] # No default (required) exchange: "" # No default (required) key: "" type: "" metadata: exclude_prefixes: [] max_in_flight: 64 ``` #### Advanced ```yml outputs: label: "" amqp_0_9: urls: [] # No default (required) exchange: "" # No default (required) exchange_declare: enabled: false type: direct durable: true arguments: "" # No default (optional) key: "" type: "" content_type: application/octet-stream content_encoding: "" correlation_id: "" reply_to: "" expiration: "" message_id: "" user_id: "" app_id: "" metadata: exclude_prefixes: [] priority: "" max_in_flight: 64 persistent: false mandatory: false immediate: false timeout: "" tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] ``` The metadata fields from each message are delivered as headers. TLS is automatically enabled when connecting to an `amqps` URL. However, you can customize [TLS settings](#tls) if required. You can use [function interpolations](../../../configuration/interpolation/#bloblang-queries) to dynamically set values for the following fields: `key`, `exchange`, and `type`. ## [](#fields)Fields ### [](#app_id)`app_id` Set an application ID for each message using a dynamic interpolated expression. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#content_encoding)`content_encoding` The content encoding attribute of each message. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#content_type)`content_type` The MIME type of each message. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `application/octet-stream` ### [](#correlation_id)`correlation_id` Set a unique correlation ID for each message using a dynamic interpolated expression to help match messages to responses. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#exchange)`exchange` The AMQP exchange to publish messages to. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#exchange_declare)`exchange_declare` Passively declares the [target exchange](#exchange) to check whether an exchange with the specified name exists and is configured correctly. If the exchange exists, then the passive declaration verifies that fields specified in this object match its properties. If the target exchange does not exist, this output creates it. **Type**: `object` ### [](#exchange_declare-arguments)`exchange_declare.arguments` Arguments for server-specific implementations of the exchange (optional). You can use arguments to configure additional parameters for exchange types that require them. **Type**: `string` ```yaml # Examples: arguments: alternate-exchange: my-ae ``` ### [](#exchange_declare-durable)`exchange_declare.durable` Whether the declared exchange is durable. **Type**: `bool` **Default**: `true` ### [](#exchange_declare-enabled)`exchange_declare.enabled` Whether to enable exchange declaration. **Type**: `bool` **Default**: `false` ### [](#exchange_declare-type)`exchange_declare.type` The type of the exchange, which determines how messages are routed to queues. > 📝 **NOTE** > > Dots (`.`) in message keys are only enforced in routing keys and message types for `topic` exchanges. **Type**: `string` **Default**: `direct` **Options**: `direct`, `fanout`, `topic`, `headers`, `x-custom` ### [](#expiration)`expiration` Set the TTL of each message in milliseconds. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#immediate)`immediate` Whether to set the immediate flag on published messages. When set to `true`, if there are no active consumers for a queue, the message is dropped instead of waiting. **Type**: `bool` **Default**: `false` ### [](#key)`key` The binding key to set for each message. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#mandatory)`mandatory` Whether to set the mandatory flag on published messages. When set to `true`, a published message that cannot be routed to any queues is returned to the sender. **Type**: `bool` **Default**: `false` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this number to improve throughput. **Type**: `int` **Default**: `64` ### [](#message_id)`message_id` Set a message ID for each message using a dynamic interpolated expression. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#metadata)`metadata` Configure which metadata values are added to messages as headers. This allows you to pass additional context information along with your messages. **Type**: `object` ### [](#metadata-exclude_prefixes)`metadata.exclude_prefixes[]` Provide a list of explicit metadata key prefixes to exclude when adding metadata to sent messages. **Type**: `array` **Default**: `[]` ### [](#persistent)`persistent` Whether to store delivered messages on disk. By default, message delivery is transient. **Type**: `bool` **Default**: `false` ### [](#priority)`priority` Set the priority of each message using a dynamic interpolated expression. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ```yaml # Examples: priority: 0 # --- priority: ${! meta("amqp_priority") } # --- priority: ${! json("doc.priority") } ``` ### [](#reply_to)`reply_to` Set the name of the queue to which responses are sent using a dynamic interpolated expression. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#timeout)`timeout` The maximum period to wait for a message acknowledgment before abandoning it and attempting a resend. If this value is not set, the system waits indefinitely. **Type**: `string` **Default**: `""` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#type)`type` A custom message type to set for each message. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#urls)`urls[]` A list of URLs to connect to. This input attempts to connect to each URL in the list, in order, until a successful connection is established. It then continues to use that URL until the connection is closed. If an item in the list contains commas, it is split into multiple URLs. Requires version 3.58.0 or later. **Type**: `array` ```yaml # Examples: urls: - "amqp://guest:guest@127.0.0.1:5672/" # --- urls: - "amqp://127.0.0.1:5672/,amqp://127.0.0.2:5672/" # --- urls: - "amqp://127.0.0.1:5672/" - "amqp://127.0.0.2:5672/" ``` ### [](#user_id)`user_id` Set the user ID to the name of the publisher. If this property is set by a publisher, its value must match the name of the user that opened the connection. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` --- # Page 123: amqp_1 **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/amqp_1.md --- # amqp\_1 --- title: amqp_1 latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/amqp_1 page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/amqp_1.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/amqp_1.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/amqp_1/)[Input](/redpanda-connect/components/inputs/amqp_1/) **Available in:** Self-Managed Sends messages to an AMQP (1.0) server. #### Common ```yml outputs: label: "" amqp_1: urls: [] # No default (optional) target_address: "" max_in_flight: 64 metadata: exclude_prefixes: [] ``` #### Advanced ```yml outputs: label: "" amqp_1: urls: [] # No default (optional) target_address: "" max_in_flight: 64 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] application_properties_map: "" # No default (optional) sasl: mechanism: none user: "" password: "" metadata: exclude_prefixes: [] content_type: opaque_binary persistent: false target_capabilities: [] # No default (optional) message_properties_to: "" # No default (optional) message_properties_message_id: "" # No default (optional) message_properties_correlation_id: "" # No default (optional) message_properties_subject: "" # No default (optional) message_properties_reply_to: "" # No default (optional) message_properties_group_id: "" # No default (optional) message_properties_group_sequence: "" # No default (optional) message_properties_reply_to_group_id: "" # No default (optional) message_properties_user_id: "" # No default (optional) message_properties_content_type: "" # No default (optional) message_properties_content_encoding: "" # No default (optional) ``` ## [](#metadata)Metadata Message metadata is added to each AMQP message as string annotations. To control which metadata keys are added, use the `metadata` configuration field. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ## [](#fields)Fields ### [](#application_properties_map)`application_properties_map` An optional Bloblang mapping that can be defined in order to set the `application-properties` on output messages. **Type**: `string` ### [](#content_type)`content_type` The content type of the message body. Set this field value to `string` to transfer each message as an AMQP string. Consider using the `string` option if you want to write UTF-8 string messages, such as JSON messages, to your data destination. **Type**: `string` **Default**: `opaque_binary` **Options**: `opaque_binary`, `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#message_properties_content_encoding)`message_properties_content_encoding` Set the content-encoding property on outgoing AMQP messages. This field supports Bloblang interpolation. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#message_properties_content_type)`message_properties_content_type` Set the content-type property on outgoing AMQP messages. This field supports Bloblang interpolation. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: message_properties_content_type: application/json # --- message_properties_content_type: text/plain; charset=utf-8 ``` ### [](#message_properties_correlation_id)`message_properties_correlation_id` Set the correlation-id property on outgoing AMQP messages. The value is auto-detected as UUID, uint64, or string. Purely numeric values are sent as uint64 on the wire. This field supports Bloblang interpolation. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: message_properties_correlation_id: ${! meta("amqp_correlation_id") } ``` ### [](#message_properties_group_id)`message_properties_group_id` Set the group-id property on outgoing AMQP messages. This field supports Bloblang interpolation. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#message_properties_group_sequence)`message_properties_group_sequence` Set the group-sequence property on outgoing AMQP messages. Must be a valid uint32 value. This field supports Bloblang interpolation. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#message_properties_message_id)`message_properties_message_id` Set the message-id property on outgoing AMQP messages. The value is auto-detected as UUID, uint64, or string. Purely numeric values are sent as uint64 on the wire. This field supports Bloblang interpolation. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: message_properties_message_id: ${! uuid_v4() } # --- message_properties_message_id: ${! meta("amqp_message_id") } ``` ### [](#message_properties_reply_to)`message_properties_reply_to` Set the reply-to property on outgoing AMQP messages. This field supports Bloblang interpolation. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#message_properties_reply_to_group_id)`message_properties_reply_to_group_id` Set the reply-to-group-id property on outgoing AMQP messages. This field supports Bloblang interpolation. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#message_properties_subject)`message_properties_subject` Set the subject property on outgoing AMQP messages. This field supports Bloblang interpolation. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#message_properties_to)`message_properties_to` The field specifies the node that is the intended destination of the message, which may differ from the node currently receiving the transfer. This field supports Bloblang interpolation. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: message_properties_to: amqp://localhost:5672/ # --- message_properties_to: ${! meta("target_address") } ``` ### [](#message_properties_user_id)`message_properties_user_id` Set the user-id property on outgoing AMQP messages. This field supports Bloblang interpolation. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#metadata-2)`metadata` Specify criteria for which metadata values are attached to messages as headers. **Type**: `object` ### [](#metadata-exclude_prefixes)`metadata.exclude_prefixes[]` Provide a list of explicit metadata key prefixes to be excluded when adding metadata to sent messages. **Type**: `array` **Default**: `[]` ### [](#persistent)`persistent` If set to true, the message will be marked as persistent, ensuring it is stored durably and not lost if an intermediary (such as a broker) restarts. By default, messages are not durable. **Type**: `bool` **Default**: `false` ### [](#sasl)`sasl` Enables SASL authentication. **Type**: `object` ### [](#sasl-mechanism)`sasl.mechanism` The SASL authentication mechanism to use. **Type**: `string` **Default**: `none` | Option | Summary | | --- | --- | | anonymous | Anonymous SASL authentication. | | none | No SASL based authentication. | | plain | Plain text SASL authentication. | ### [](#sasl-password)`sasl.password` A SASL plain text password. It is recommended that you use environment variables to populate this field. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: ${PASSWORD} ``` ### [](#sasl-user)`sasl.user` A SASL plain text username. It is recommended that you use environment variables to populate this field. **Type**: `string` **Default**: `""` ```yaml # Examples: user: ${USER} ``` ### [](#target_address)`target_address` The target address to write to. When left empty, the output uses the Anonymous Terminus pattern where the destination is specified per-message using `message_properties_to`. **Type**: `string` **Default**: `""` ```yaml # Examples: target_address: /foo # --- target_address: queue:/bar # --- target_address: topic:/baz # --- target_address: ``` ### [](#target_capabilities)`target_capabilities[]` Lists the extension capabilities the sender desires from the target, such as support for queues, topics, durability, sharing, or temporary destinations. **Type**: `array` ```yaml # Examples: target_capabilities: - queue # --- target_capabilities: - topic # --- target_capabilities: - queue - topic ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. The first URL to successfully establish a connection will be used until the connection is closed. If an item of the list contains commas it will be expanded into multiple URLs. Requires version 4.23.0 or later. **Type**: `array` ```yaml # Examples: urls: - "amqp://guest:guest@127.0.0.1:5672/" # --- urls: - "amqp://127.0.0.1:5672/,amqp://127.0.0.2:5672/" # --- urls: - "amqp://127.0.0.1:5672/" - "amqp://127.0.0.2:5672/" ``` --- # Page 124: aws_dynamodb **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/aws_dynamodb.md --- # aws\_dynamodb --- title: aws_dynamodb latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/aws_dynamodb page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/aws_dynamodb.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/aws_dynamodb.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/aws_dynamodb/)[Cache](/redpanda-connect/components/caches/aws_dynamodb/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/aws_dynamodb/ "View the Cloud version of this component"), Self-Managed Inserts items into a DynamoDB table. Introduced in version 3.36.0. #### Common ```yml outputs: label: "" aws_dynamodb: table: "" # No default (required) string_columns: {} json_map_columns: {} max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" aws_dynamodb: table: "" # No default (required) string_columns: {} json_map_columns: {} ttl: "" ttl_key: "" max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) max_retries: 3 backoff: initial_interval: 1s max_interval: 5s max_elapsed_time: 30s ``` The field `string_columns` is a map of column names to string values, where the values are [function interpolated](../../../configuration/interpolation/#bloblang-queries) per message of a batch. This allows you to populate string columns of an item by extracting fields within the document payload or metadata like follows: ```yml string_columns: id: ${!json("id")} title: ${!json("body.title")} topic: ${!meta("kafka_topic")} full_content: ${!content()} ``` The field `json_map_columns` is a map of column names to json paths, where the [dot path](../../../configuration/field_paths/) is extracted from each document and converted into a map value. Both an empty path and the path `.` are interpreted as the root of the document. This allows you to populate map columns of an item like follows: ```yml json_map_columns: user: path.to.user whole_document: . ``` A column name can be empty: ```yml json_map_columns: "": . ``` In which case the top level document fields will be written at the root of the item, potentially overwriting previously defined column values. If a path is not found within a document the column will not be populated. ## [](#credentials)Credentials By default Redpanda Connect will use a shared credentials file when connecting to AWS services. It’s also possible to set them explicitly at the component level, allowing you to transfer data across accounts. You can find out more in [Amazon Web Services](../../../guides/cloud/aws/). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](../../../configuration/batching/). ## [](#fields)Fields ### [](#backoff)`backoff` Control time intervals between retry attempts. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` The initial period to wait between retry attempts. The retry interval increases for each failed attempt, up to the `backoff.max_interval` value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `1s` ### [](#backoff-max_elapsed_time)`backoff.max_elapsed_time` The maximum period to wait before retry attempts are abandoned. If zero then no limit is used. **Type**: `string` **Default**: `30s` ### [](#backoff-max_interval)`backoff.max_interval` The maximum period to wait between retry attempts. **Type**: `string` **Default**: `5s` ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#credentials-2)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](../../../guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#json_map_columns)`json_map_columns` A map of column keys to [field paths](../../../configuration/field_paths/) pointing to value data within messages. **Type**: `string` **Default**: `{}` ```yaml # Examples: json_map_columns: user: path.to.user whole_document: . # --- json_map_columns: "": . ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#max_retries)`max_retries` The maximum number of retries before giving up on the request. If set to zero there is no discrete limit. **Type**: `int` **Default**: `3` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#string_columns)`string_columns` A map of column keys to string values to store. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `{}` ```yaml # Examples: string_columns: full_content: ${!content()} id: ${!json("id")} title: ${!json("body.title")} topic: ${!meta("kafka_topic")} ``` ### [](#table)`table` The table to store messages in. **Type**: `string` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#ttl)`ttl` An optional TTL to set for items, calculated from the moment the message is sent. **Type**: `string` **Default**: `""` ### [](#ttl_key)`ttl_key` The column key to place the TTL value within. **Type**: `string` **Default**: `""` --- # Page 125: aws_kinesis_firehose **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/aws_kinesis_firehose.md --- # aws\_kinesis\_firehose --- title: aws_kinesis_firehose latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/aws_kinesis_firehose page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/aws_kinesis_firehose.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/aws_kinesis_firehose.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/aws_kinesis_firehose/ "View the Cloud version of this component"), Self-Managed Sends messages to a Kinesis Firehose delivery stream. Introduced in version 3.36.0. #### Common ```yml outputs: label: "" aws_kinesis_firehose: stream: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" aws_kinesis_firehose: stream: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) max_retries: 0 backoff: initial_interval: 1s max_interval: 5s max_elapsed_time: 30s ``` ## [](#credentials)Credentials By default Redpanda Connect will use a shared credentials file when connecting to AWS services. It’s also possible to set them explicitly at the component level, allowing you to transfer data across accounts. You can find out more in [Amazon Web Services](../../../guides/cloud/aws/). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](../../../configuration/batching/). ## [](#fields)Fields ### [](#backoff)`backoff` Control time intervals between retry attempts. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` The initial period to wait between retry attempts. The retry interval increases for each failed attempt, up to the `backoff.max_interval` value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `1s` ### [](#backoff-max_elapsed_time)`backoff.max_elapsed_time` The maximum period to wait before retry attempts are abandoned. If zero then no limit is used. **Type**: `string` **Default**: `30s` ### [](#backoff-max_interval)`backoff.max_interval` The maximum period to wait between retry attempts. **Type**: `string` **Default**: `5s` ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#credentials-2)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](../../../guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#max_retries)`max_retries` The maximum number of retries before giving up on the request. If set to zero there is no discrete limit. **Type**: `int` **Default**: `0` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#stream)`stream` The stream to publish messages to. **Type**: `string` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` --- # Page 126: aws_kinesis **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/aws_kinesis.md --- # aws\_kinesis --- title: aws_kinesis latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/aws_kinesis page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/aws_kinesis.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/aws_kinesis.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/aws_kinesis/)[Input](/redpanda-connect/components/inputs/aws_kinesis/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/aws_kinesis/ "View the Cloud version of this component"), Self-Managed Sends messages to a Kinesis stream. Introduced in version 3.36.0. #### Common ```yml outputs: label: "" aws_kinesis: stream: "" # No default (required) partition_key: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" aws_kinesis: stream: "" # No default (required) partition_key: "" # No default (required) hash_key: "" # No default (optional) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) max_retries: 0 backoff: initial_interval: 1s max_interval: 5s max_elapsed_time: 30s ``` Both the `partition_key`(required) and `hash_key` (optional) fields can be dynamically set using function interpolations described [here](../../../configuration/interpolation/#bloblang-queries). When sending batched messages the interpolations are performed per message part. ## [](#credentials)Credentials By default Redpanda Connect will use a shared credentials file when connecting to AWS services. It’s also possible to set them explicitly at the component level, allowing you to transfer data across accounts. You can find out more in [Amazon Web Services](../../../guides/cloud/aws/). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](../../../configuration/batching/). ## [](#fields)Fields ### [](#backoff)`backoff` Control time intervals between retry attempts. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` The initial period to wait between retry attempts. The retry interval increases for each failed attempt, up to the `backoff.max_interval` value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `1s` ### [](#backoff-max_elapsed_time)`backoff.max_elapsed_time` The maximum period to wait before retry attempts are abandoned. If zero then no limit is used. **Type**: `string` **Default**: `30s` ### [](#backoff-max_interval)`backoff.max_interval` The maximum period to wait between retry attempts. **Type**: `string` **Default**: `5s` ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#credentials-2)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](../../../guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#hash_key)`hash_key` A optional hash key for partitioning messages. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of parallel message batches to have in flight at any given time. **Type**: `int` **Default**: `64` ### [](#max_retries)`max_retries` The maximum number of retries before giving up on the request. If set to zero there is no discrete limit. **Type**: `int` **Default**: `0` ### [](#partition_key)`partition_key` A required key for partitioning messages. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#stream)`stream` The stream to publish messages to. Streams can either be specified by their name or full ARN. **Type**: `string` ```yaml # Examples: stream: foo # --- stream: arn:aws:kinesis:*:111122223333:stream/my-stream ``` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` --- # Page 127: aws_s3 **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/aws_s3.md --- # aws\_s3 --- title: aws_s3 latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/aws_s3 page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/aws_s3.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/aws_s3.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-03-19" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/aws_s3/)[Cache](/redpanda-connect/components/caches/aws_s3/)[Input](/redpanda-connect/components/inputs/aws_s3/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/aws_s3/ "View the Cloud version of this component"), Self-Managed Uploads messages to an Amazon S3 bucket as objects, using the path specified in the `path` field. Introduced in version 3.36.0. #### Common ```yml outputs: label: "" aws_s3: bucket: "" # No default (required) path: ${!counter()}-${!timestamp_unix_nano()}.txt tags: {} content_type: application/octet-stream metadata: exclude_prefixes: [] max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" aws_s3: bucket: "" # No default (required) path: ${!counter()}-${!timestamp_unix_nano()}.txt tags: {} content_type: application/octet-stream content_encoding: "" cache_control: "" content_disposition: "" content_language: "" website_redirect_location: "" metadata: exclude_prefixes: [] storage_class: STANDARD kms_key_id: "" checksum_algorithm: "" server_side_encryption: "" force_path_style_urls: false max_in_flight: 64 timeout: 5s object_canned_acl: private batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` To use a different path for each object, use [function interpolation](../../../configuration/interpolation/#bloblang-queries), which is evaluated for each message in a batch. ## [](#metadata)Metadata Redpanda Connect sends metadata fields as headers. To mutate or remove these values, see the [metadata docs](../../../configuration/metadata/). ## [](#tags)Tags The `tags` field accepts key/value pairs to attach to objects as tags, and the values support [interpolation functions](../../../configuration/interpolation/#bloblang-queries): ```yaml output: aws_s3: bucket: TODO path: ${!counter()}-${!timestamp_unix_nano()}.tar.gz tags: Key1: Value1 Timestamp: ${!meta("Timestamp")} ``` ## [](#credentials)Credentials By default, Redpanda Connect uses a shared credentials file when connecting to AWS services. You can also set credentials explicitly at the component level to transfer data across accounts. You can find out more in [AWS credentials](../../../guides/cloud/aws/). ## [](#batching)Batching It’s common to want to upload messages to S3 as batched archives. The easiest way to do this is to batch your messages at the output level and join the batch of messages with an [`archive`](../../processors/archive/) or [`compress`](../../processors/compress/) processor. For example, the following configuration uploads messages as a `.tar.gz` archive of documents: ```yaml output: aws_s3: bucket: TODO path: ${!counter()}-${!timestamp_unix_nano()}.tar.gz batching: count: 100 period: 10s processors: - archive: format: tar - compress: algorithm: gzip ``` This configuration uploads JSON documents as a single large document containing an array of objects: ```yaml output: aws_s3: bucket: TODO path: ${!counter()}-${!timestamp_unix_nano()}.json batching: count: 100 processors: - archive: format: json_array ``` ## [](#bucket-name-format)Bucket name format The `bucket` field accepts a bucket name only, not an ARN. For example, use `my-bucket`, not `arn:aws:s3:::my-bucket`. ## [](#s3-compatible-storage)S3-compatible storage The `endpoint` and `force_path_style_urls` fields let you connect to S3-compatible storage services such as Cloudflare R2, MinIO, or DigitalOcean Spaces. For Cloudflare R2, set `endpoint` to your account endpoint URL and enable `force_path_style_urls`: ```yaml output: aws_s3: bucket: r2-bucket path: ${!uuid_v4()}.json endpoint: https://.r2.cloudflarestorage.com force_path_style_urls: true region: auto credentials: id: secret: ``` Find your account ID in the Cloudflare dashboard under **R2 > Overview > Account Details**. Generate API credentials under **R2 > Manage R2 API Tokens**. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ## [](#fields)Fields ### [](#batching-2)`batching` Configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#bucket)`bucket` The bucket to upload messages to. **Type**: `string` ### [](#cache_control)`cache_control` The cache control to set for each object. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#checksum_algorithm)`checksum_algorithm` The algorithm used to validate each object during its upload to the Amazon S3 bucket. **Type**: `string` **Default**: `""` **Options**: `CRC32`, `CRC32C`, `SHA1`, `SHA256` ### [](#content_disposition)`content_disposition` The content disposition to set for each object. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#content_encoding)`content_encoding` An optional content encoding to set for each object. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#content_language)`content_language` The content language to set for each object. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#content_type)`content_type` The content type to set for each object. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `application/octet-stream` ### [](#credentials-2)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](../../../guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#force_path_style_urls)`force_path_style_urls` Forces the client API to use path style URLs, which helps when connecting to custom endpoints. **Type**: `bool` **Default**: `false` ### [](#kms_key_id)`kms_key_id` An optional server-side encryption key. **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#metadata-2)`metadata` Specify criteria for which metadata values are attached to objects as headers. **Type**: `object` ### [](#metadata-exclude_prefixes)`metadata.exclude_prefixes[]` Provide a list of explicit metadata key prefixes to be excluded when adding metadata to sent messages. **Type**: `array` **Default**: `[]` ### [](#object_canned_acl)`object_canned_acl` The object canned ACL value. **Type**: `string` **Default**: `private` **Options**: `private`, `public-read`, `public-read-write`, `authenticated-read`, `aws-exec-read`, `bucket-owner-read`, `bucket-owner-full-control` ### [](#path)`path` The path of each message to upload. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `${!counter()}-${!timestamp_unix_nano()}.txt` ```yaml # Examples: path: ${!counter()}-${!timestamp_unix_nano()}.txt # --- path: ${!meta("kafka_key")}.json # --- path: ${!json("doc.namespace")}/${!json("doc.id")}.json ``` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#server_side_encryption)`server_side_encryption` An optional server-side encryption algorithm. Requires version 3.63.0 or later. **Type**: `string` **Default**: `""` ### [](#storage_class)`storage_class` The storage class to set for each object. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `STANDARD` **Options**: `STANDARD`, `REDUCED_REDUNDANCY`, `GLACIER`, `STANDARD_IA`, `ONEZONE_IA`, `INTELLIGENT_TIERING`, `DEEP_ARCHIVE` ### [](#tags-2)`tags` Key/value pairs to store with the object as tags. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `{}` ```yaml # Examples: tags: Key1: Value1 Timestamp: ${!meta("Timestamp")} ``` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` The maximum period to wait on an upload before abandoning it and reattempting. **Type**: `string` **Default**: `5s` ### [](#website_redirect_location)`website_redirect_location` The website redirect location to set for each object. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` --- # Page 128: aws_sns **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/aws_sns.md --- # aws\_sns --- title: aws_sns latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/aws_sns page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/aws_sns.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/aws_sns.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/aws_sns/ "View the Cloud version of this component"), Self-Managed Sends messages to an AWS SNS topic. Introduced in version 3.36.0. #### Common ```yml outputs: label: "" aws_sns: topic_arn: "" # No default (required) message_group_id: "" # No default (optional) message_deduplication_id: "" # No default (optional) subject: "" # No default (optional) max_in_flight: 64 metadata: exclude_prefixes: [] ``` #### Advanced ```yml outputs: label: "" aws_sns: topic_arn: "" # No default (required) message_group_id: "" # No default (optional) message_deduplication_id: "" # No default (optional) subject: "" # No default (optional) max_in_flight: 64 metadata: exclude_prefixes: [] timeout: 5s region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` ## [](#credentials)Credentials By default Redpanda Connect will use a shared credentials file when connecting to AWS services. It’s also possible to set them explicitly at the component level, allowing you to transfer data across accounts. You can find out more in [Amazon Web Services](../../../guides/cloud/aws/). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ## [](#fields)Fields ### [](#credentials-2)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](../../../guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#message_deduplication_id)`message_deduplication_id` An optional deduplication ID to set for messages. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). Requires version 3.60.0 or later. **Type**: `string` ### [](#message_group_id)`message_group_id` An optional group ID to set for messages. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). Requires version 3.60.0 or later. **Type**: `string` ### [](#metadata)`metadata` Specify criteria for which metadata values are sent as headers. Requires version 3.60.0 or later. **Type**: `object` ### [](#metadata-exclude_prefixes)`metadata.exclude_prefixes[]` Provide a list of explicit metadata key prefixes to be excluded when adding metadata to sent messages. **Type**: `array` **Default**: `[]` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#subject)`subject` An optional subject to set for messages. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` The maximum period to wait on an upload before abandoning it and reattempting. **Type**: `string` **Default**: `5s` ### [](#topic_arn)`topic_arn` The topic to publish to. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 129: aws_sqs **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/aws_sqs.md --- # aws\_sqs --- title: aws_sqs latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/aws_sqs page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/aws_sqs.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/aws_sqs.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/aws_sqs/)[Input](/redpanda-connect/components/inputs/aws_sqs/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/aws_sqs/ "View the Cloud version of this component"), Self-Managed Sends messages to an SQS queue. Introduced in version 3.36.0. #### Common ```yml outputs: label: "" aws_sqs: url: "" # No default (required) message_group_id: "" # No default (optional) message_deduplication_id: "" # No default (optional) delay_seconds: "" # No default (optional) max_in_flight: 64 metadata: exclude_prefixes: [] batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" aws_sqs: url: "" # No default (required) message_group_id: "" # No default (optional) message_deduplication_id: "" # No default (optional) delay_seconds: "" # No default (optional) max_in_flight: 64 metadata: exclude_prefixes: [] batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_records_per_request: 10 region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) max_retries: 0 backoff: initial_interval: 1s max_interval: 5s max_elapsed_time: 30s ``` Metadata values are sent along with the payload as attributes with the data type String. If the number of metadata values in a message exceeds the message attribute limit (10) then the top ten keys ordered alphabetically will be selected. The fields `message_group_id`, `message_deduplication_id` and `delay_seconds` can be set dynamically using [function interpolations](../../../configuration/interpolation/#bloblang-queries), which are resolved individually for each message of a batch. ## [](#credentials)Credentials By default Redpanda Connect will use a shared credentials file when connecting to AWS services. It’s also possible to set them explicitly at the component level, allowing you to transfer data across accounts. You can find out more in [Amazon Web Services](../../../guides/cloud/aws/). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](../../../configuration/batching/). ## [](#fields)Fields ### [](#backoff)`backoff` Control time intervals between retry attempts. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` The initial period to wait between retry attempts. The retry interval increases for each failed attempt, up to the `backoff.max_interval` value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `1s` ### [](#backoff-max_elapsed_time)`backoff.max_elapsed_time` The maximum period to wait before retry attempts are abandoned. If zero then no limit is used. **Type**: `string` **Default**: `30s` ### [](#backoff-max_interval)`backoff.max_interval` The maximum period to wait between retry attempts. **Type**: `string` **Default**: `5s` ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#credentials-2)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](../../../guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#delay_seconds)`delay_seconds` An optional delay time in seconds for message. Value between 0 and 900 This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of parallel message batches to have in flight at any given time. **Type**: `int` **Default**: `64` ### [](#max_records_per_request)`max_records_per_request` The maximum number of records delivered in a single SQS request. Enter only values from `0` to `10`. **Type**: `int` **Default**: `10` ### [](#max_retries)`max_retries` The maximum number of retries before giving up on the request. If set to zero there is no discrete limit. **Type**: `int` **Default**: `0` ### [](#message_deduplication_id)`message_deduplication_id` An optional deduplication ID to set for messages. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#message_group_id)`message_group_id` An optional group ID to set for messages. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#metadata)`metadata` Specify criteria for which metadata values are sent as headers. **Type**: `object` ### [](#metadata-exclude_prefixes)`metadata.exclude_prefixes[]` Provide a list of explicit metadata key prefixes to be excluded when adding metadata to sent messages. **Type**: `array` **Default**: `[]` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#url)`url` The URL of the target SQS queue. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 130: azure_blob_storage **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/azure_blob_storage.md --- # azure\_blob\_storage --- title: azure_blob_storage latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/azure_blob_storage page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/azure_blob_storage.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/azure_blob_storage.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/azure_blob_storage/)[Input](/redpanda-connect/components/inputs/azure_blob_storage/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/azure_blob_storage/ "View the Cloud version of this component"), Self-Managed Sends message parts as objects to an Azure Blob Storage Account container. Each object is uploaded with the filename specified with the `container` field. Introduced in version 3.36.0. #### Common ```yml outputs: label: "" azure_blob_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" container: "" # No default (required) path: ${!counter()}-${!timestamp_unix_nano()}.txt max_in_flight: 64 ``` #### Advanced ```yml outputs: label: "" azure_blob_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" container: "" # No default (required) path: ${!counter()}-${!timestamp_unix_nano()}.txt blob_type: BLOCK public_access_level: PRIVATE max_in_flight: 64 ``` In order to have a different path for each object you should use function interpolations described [here](../../../configuration/interpolation/#bloblang-queries), which are calculated per message of a batch. Supports multiple authentication methods but only one of the following is required: - `storage_connection_string` - `storage_account` and `storage_access_key` - `storage_account` and `storage_sas_token` - `storage_account` to access via [DefaultAzureCredential](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#DefaultAzureCredential) If multiple are set then the `storage_connection_string` is given priority. If the `storage_connection_string` does not contain the `AccountName` parameter, please specify it in the `storage_account` field. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ## [](#fields)Fields ### [](#blob_type)`blob_type` Block and Append blobs are comprized of blocks, and each blob can support up to 50,000 blocks. The default value is ``"`BLOCK`"``.\` This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `BLOCK` **Options**: `BLOCK`, `APPEND` ### [](#container)`container` The container for uploading the messages to. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: container: messages-${!timestamp("2006")} ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#path)`path` The path of each message to upload. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `${!counter()}-${!timestamp_unix_nano()}.txt` ```yaml # Examples: path: ${!counter()}-${!timestamp_unix_nano()}.json # --- path: ${!meta("kafka_key")}.json # --- path: ${!json("doc.namespace")}/${!json("doc.id")}.json ``` ### [](#public_access_level)`public_access_level` The container’s public access level. The default value is `PRIVATE`. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `PRIVATE` **Options**: `PRIVATE`, `BLOB`, `CONTAINER` ### [](#storage_access_key)`storage_access_key` The storage account access key. This field is ignored if `storage_connection_string` is set. **Type**: `string` **Default**: `""` ### [](#storage_account)`storage_account` The storage account to access. This field is ignored if `storage_connection_string` is set. **Type**: `string` **Default**: `""` ### [](#storage_connection_string)`storage_connection_string` A storage account connection string. This field is required if `storage_account` and `storage_access_key` / `storage_sas_token` are not set. **Type**: `string` **Default**: `""` ### [](#storage_sas_token)`storage_sas_token` The storage account SAS token. This field is ignored if `storage_connection_string` or `storage_access_key` are set. **Type**: `string` **Default**: `""` --- # Page 131: azure_cosmosdb **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/azure_cosmosdb.md --- # azure\_cosmosdb --- title: azure_cosmosdb latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/azure_cosmosdb page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/azure_cosmosdb.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/azure_cosmosdb.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/azure_cosmosdb/)[Input](/redpanda-connect/components/inputs/azure_cosmosdb/)[Processor](/redpanda-connect/components/processors/azure_cosmosdb/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/azure_cosmosdb/ "View the Cloud version of this component"), Self-Managed Creates or updates messages as JSON documents in [Azure CosmosDB](https://learn.microsoft.com/en-us/azure/cosmos-db/introduction). Introduced in version v4.25.0. #### Common ```yml outputs: label: "" azure_cosmosdb: endpoint: "" # No default (optional) account_key: "" # No default (optional) connection_string: "" # No default (optional) database: "" # No default (required) container: "" # No default (required) partition_keys_map: "" # No default (required) operation: Create item_id: "" # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_in_flight: 64 ``` #### Advanced ```yml outputs: label: "" azure_cosmosdb: endpoint: "" # No default (optional) account_key: "" # No default (optional) connection_string: "" # No default (optional) database: "" # No default (required) container: "" # No default (required) partition_keys_map: "" # No default (required) operation: Create patch_operations: [] # No default (optional) patch_condition: "" # No default (optional) auto_id: true item_id: "" # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_in_flight: 64 ``` When creating documents, each message must have the `id` property (case-sensitive) set (or use `auto_id: true`). It is the unique name that identifies the document, that is, no two documents share the same `id` within a logical partition. The `id` field must not exceed 255 characters. [See details](https://learn.microsoft.com/en-us/rest/api/cosmos-db/documents). The `partition_keys` field must resolve to the same value(s) across the entire message batch. ## [](#credentials)Credentials You can use one of the following authentication mechanisms: - Set the `endpoint` field and the `account_key` field - Set only the `endpoint` field to use [DefaultAzureCredential](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#DefaultAzureCredential) - Set the `connection_string` field ## [](#batching)Batching CosmosDB limits the maximum batch size to 100 messages and the payload must not exceed 2MB ([details here](https://learn.microsoft.com/en-us/azure/cosmos-db/concepts-limits#per-request-limits)). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](../../../configuration/batching/). ## [](#examples)Examples ### [](#create-documents)Create documents Create new documents in the `blobfish` container with partition key `/habitat`. ```yaml output: azure_cosmosdb: endpoint: http://localhost:8080 account_key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw== database: blobbase container: blobfish partition_keys_map: root = json("habitat") operation: Create ``` ### [](#patch-documents)Patch documents Execute the Patch operation on documents from the `blobfish` container. ```yaml output: azure_cosmosdb: endpoint: http://localhost:8080 account_key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw== database: testdb container: blobfish partition_keys_map: root = json("habitat") item_id: ${! json("id") } operation: Patch patch_operations: # Add a new /diet field - operation: Add path: /diet value_map: root = json("diet") # Remove the first location from the /locations array field - operation: Remove path: /locations/0 # Add new location at the end of the /locations array field - operation: Add path: /locations/- value_map: root = "Challenger Deep" ``` ## [](#fields)Fields ### [](#account_key)`account_key` Account key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ```yaml # Examples: account_key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw== ``` ### [](#auto_id)`auto_id` Automatically set the item `id` field to a random UUID v4. If the `id` field is already set, then it will not be overwritten. Setting this to `false` can improve performance, since the messages will not have to be parsed. **Type**: `bool` **Default**: `true` ### [](#batching-2)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#connection_string)`connection_string` Connection string. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ```yaml # Examples: connection_string: AccountEndpoint=https://localhost:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==; ``` ### [](#container)`container` Container. **Type**: `string` ```yaml # Examples: container: testcontainer ``` ### [](#database)`database` Database. **Type**: `string` ```yaml # Examples: database: testdb ``` ### [](#endpoint)`endpoint` CosmosDB endpoint. **Type**: `string` ```yaml # Examples: endpoint: https://localhost:8081 ``` ### [](#item_id)`item_id` ID of item to replace or delete. Only used by the Replace and Delete operations This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: item_id: ${! json("id") } ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#operation)`operation` Operation. **Type**: `string` **Default**: `Create` | Option | Summary | | --- | --- | | Create | Create operation. | | Delete | Delete operation. | | Patch | Patch operation. | | Replace | Replace operation. | | Upsert | Upsert operation. | ### [](#partition_keys_map)`partition_keys_map` A [Bloblang mapping](../../../guides/bloblang/about/) which should evaluate to a single partition key value or an array of partition key values of type string, integer or boolean. Currently, hierarchical partition keys are not supported so only one value may be provided. **Type**: `string` ```yaml # Examples: partition_keys_map: root = "blobfish" # --- partition_keys_map: root = 41 # --- partition_keys_map: root = true # --- partition_keys_map: root = null # --- partition_keys_map: root = json("blobfish").depth ``` ### [](#patch_condition)`patch_condition` Patch operation condition. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: patch_condition: from c where not is_defined(c.blobfish) ``` ### [](#patch_operations)`patch_operations[]` Patch operations to be performed when `operation: Patch` . **Type**: `object` ### [](#patch_operations-operation)`patch_operations[].operation` Operation. **Type**: `string` **Default**: `Add` | Option | Summary | | --- | --- | | Add | Add patch operation. | | Increment | Increment patch operation. | | Remove | Remove patch operation. | | Replace | Replace patch operation. | | Set | Set patch operation. | ### [](#patch_operations-path)`patch_operations[].path` Path. **Type**: `string` ```yaml # Examples: path: /foo/bar/baz ``` ### [](#patch_operations-value_map)`patch_operations[].value_map` A [Bloblang mapping](../../../guides/bloblang/about/) which should evaluate to a value of any type that is supported by CosmosDB. **Type**: `string` ```yaml # Examples: value_map: root = "blobfish" # --- value_map: root = 41 # --- value_map: root = true # --- value_map: root = json("blobfish").depth # --- value_map: root = [1, 2, 3] ``` ## [](#cosmosdb-emulator)CosmosDB emulator If you wish to run the CosmosDB emulator that is referenced in the documentation [here](https://learn.microsoft.com/en-us/azure/cosmos-db/linux-emulator), the following Docker command should do the trick: ```bash > docker run --rm -it -p 8081:8081 --name=cosmosdb -e AZURE_COSMOS_EMULATOR_PARTITION_COUNT=10 -e AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE=false mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator ``` Note: `AZURE_COSMOS_EMULATOR_PARTITION_COUNT` controls the number of partitions that will be supported by the emulator. The bigger the value, the longer it takes for the container to start up. Additionally, instead of installing the container self-signed certificate which is exposed via `[https://localhost:8081/_explorer/emulator.pem](https://localhost:8081/_explorer/emulator.pem)`, you can run [mitmproxy](https://mitmproxy.org/) like so: ```bash > mitmproxy -k --mode "reverse:https://localhost:8081" ``` Then you can access the CosmosDB UI via `[http://localhost:8080/_explorer/index.html](http://localhost:8080/_explorer/index.html)` and use `[http://localhost:8080](http://localhost:8080)` as the CosmosDB endpoint. --- # Page 132: azure_data_lake_gen2 **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/azure_data_lake_gen2.md --- # azure\_data\_lake\_gen2 --- title: azure_data_lake_gen2 latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/azure_data_lake_gen2 page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/azure_data_lake_gen2.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/azure_data_lake_gen2.adoc page-git-created-date: "2024-11-05" page-git-modified-date: "2026-01-13" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/azure_data_lake_gen2/ "View the Cloud version of this component"), Self-Managed Sends message parts as files to an [Azure Data Lake Gen2](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction) file system. Each file is uploaded with the file name specified in the `path` field. Introduced in version 4.38.0. ```yml outputs: label: "" azure_data_lake_gen2: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" filesystem: "" # No default (required) path: ${!counter()}-${!timestamp_unix_nano()}.txt max_in_flight: 64 ``` To specify a different [`path` value](#path) (file name) for each file, use [function interpolations](../../../configuration/interpolation/#bloblang-queries). Function interpolations are calculated for each message in a batch. ## [](#authentication-methods)Authentication methods This output supports multiple authentication methods. You must configure at least one method from the following list: - `storage_connection_string` - `storage_account` and `storage_access_key` - `storage_account` and `storage_sas_token` - `storage_account` to access using [DefaultAzureCredential](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#DefaultAzureCredential) If you configure multiple authentication methods, the `storage_connection_string` takes precedence. ## [](#performance)Performance Sends multiple messages in flight in parallel for improved performance. You can tune the number of in flight messages (or message batches) with the field `max_in_flight`. ## [](#fields)Fields ### [](#filesystem)`filesystem` The name of the data lake storage file system you want to upload messages to. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: filesystem: messages-${!timestamp("2006")} ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this number to improve throughput until performance plateaus. **Type**: `int` **Default**: `64` ### [](#path)`path` The path (file name) of each message to upload to the data lake storage file system. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `${!counter()}-${!timestamp_unix_nano()}.txt` ```yaml # Examples: path: ${!counter()}-${!timestamp_unix_nano()}.json # --- path: ${!meta("kafka_key")}.json # --- path: ${!json("doc.namespace")}/${!json("doc.id")}.json ``` ### [](#storage_access_key)`storage_access_key` The access key for the storage account. Use this field along with `storage_account` for authentication. This field is ignored when the `storage_connection_string` field is populated. **Type**: `string` **Default**: `""` ### [](#storage_account)`storage_account` The storage account to access. This field is ignored when the `storage_connection_string` field is populated. **Type**: `string` **Default**: `""` ### [](#storage_connection_string)`storage_connection_string` The connection string for the storage account. You must enter a value for this field if no other authentication method is specified. > 📝 **NOTE** > > If the `storage_connection_string` field does not contain the `AccountName` parameter value, specify it in the `storage_account` field. **Type**: `string` **Default**: `""` ### [](#storage_sas_token)`storage_sas_token` The SAS token for the storage account. Use this field along with `storage_account` for authentication. This field is ignored when either the `storage_connection_string` or `storage_access_key` fields are populated. **Type**: `string` **Default**: `""` --- # Page 133: azure_queue_storage **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/azure_queue_storage.md --- # azure\_queue\_storage --- title: azure_queue_storage latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/azure_queue_storage page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/azure_queue_storage.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/azure_queue_storage.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/azure_queue_storage/)[Input](/redpanda-connect/components/inputs/azure_queue_storage/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/azure_queue_storage/ "View the Cloud version of this component"), Self-Managed Sends messages to an Azure Storage Queue. Introduced in version 3.36.0. #### Common ```yml outputs: label: "" azure_queue_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" queue_name: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" azure_queue_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" queue_name: "" # No default (required) ttl: "" max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` Only one authentication method is required, `storage_connection_string` or `storage_account` and `storage_access_key`. If both are set then the `storage_connection_string` is given priority. In order to set the `queue_name` you can use function interpolations described [here](../../../configuration/interpolation/#bloblang-queries), which are calculated per message of a batch. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](../../../configuration/batching/). ## [](#fields)Fields ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#max_in_flight)`max_in_flight` The maximum number of parallel message batches to have in flight at any given time. **Type**: `int` **Default**: `64` ### [](#queue_name)`queue_name` The name of the target Queue Storage queue. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#storage_access_key)`storage_access_key` The storage account access key. This field is ignored if `storage_connection_string` is set. **Type**: `string` **Default**: `""` ### [](#storage_account)`storage_account` The storage account to access. This field is ignored if `storage_connection_string` is set. **Type**: `string` **Default**: `""` ### [](#storage_connection_string)`storage_connection_string` A storage account connection string. This field is required if `storage_account` and `storage_access_key` / `storage_sas_token` are not set. **Type**: `string` **Default**: `""` ### [](#storage_sas_token)`storage_sas_token` The storage account SAS token. This field is ignored if `storage_connection_string` or `storage_access_key` are set. **Type**: `string` **Default**: `""` ### [](#ttl)`ttl` The TTL of each individual message as a duration string. Defaults to 0, meaning no retention period is set This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ```yaml # Examples: ttl: 60s # --- ttl: 5m # --- ttl: 36h ``` --- # Page 134: azure_table_storage **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/azure_table_storage.md --- # azure\_table\_storage --- title: azure_table_storage latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/azure_table_storage page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/azure_table_storage.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/azure_table_storage.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/azure_table_storage/)[Input](/redpanda-connect/components/inputs/azure_table_storage/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/azure_table_storage/ "View the Cloud version of this component"), Self-Managed Stores messages in an Azure Table Storage table. Introduced in version 3.36.0. #### Common ```yml outputs: label: "" azure_table_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" table_name: "" # No default (required) partition_key: "" row_key: "" properties: {} max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" azure_table_storage: storage_account: "" storage_access_key: "" storage_connection_string: "" storage_sas_token: "" table_name: "" # No default (required) partition_key: "" row_key: "" properties: {} transaction_type: INSERT max_in_flight: 64 timeout: 5s batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` Only one authentication method is required, `storage_connection_string` or `storage_account` and `storage_access_key`. If both are set then the `storage_connection_string` is given priority. In order to set the `table_name`, `partition_key` and `row_key` you can use function interpolations described [here](../../../configuration/interpolation/#bloblang-queries), which are calculated per message of a batch. If the `properties` are not set in the config, all the `json` fields are marshalled and stored in the table, which will be created if it does not exist. The `object` and `array` fields are marshaled as strings. e.g.: The JSON message: ```json { "foo": 55, "bar": { "baz": "a", "bez": "b" }, "diz": ["a", "b"] } ``` Will store in the table the following properties: ```yml foo: '55' bar: '{ "baz": "a", "bez": "b" }' diz: '["a", "b"]' ``` It’s also possible to use function interpolations to get or transform the properties values, e.g.: ```yml properties: device: '${! json("device") }' timestamp: '${! json("timestamp") }' ``` ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](../../../configuration/batching/). ## [](#fields)Fields ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#max_in_flight)`max_in_flight` The maximum number of parallel message batches to have in flight at any given time. **Type**: `int` **Default**: `64` ### [](#partition_key)`partition_key` The partition key. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ```yaml # Examples: partition_key: ${! json("date") } ``` ### [](#properties)`properties` A map of properties to store into the table. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `{}` ### [](#row_key)`row_key` The row key. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ```yaml # Examples: row_key: ${! json("device")}-${!uuid_v4() } ``` ### [](#storage_access_key)`storage_access_key` The storage account access key. This field is ignored if `storage_connection_string` is set. **Type**: `string` **Default**: `""` ### [](#storage_account)`storage_account` The storage account to access. This field is ignored if `storage_connection_string` is set. **Type**: `string` **Default**: `""` ### [](#storage_connection_string)`storage_connection_string` A storage account connection string. This field is required if `storage_account` and `storage_access_key` / `storage_sas_token` are not set. **Type**: `string` **Default**: `""` ### [](#storage_sas_token)`storage_sas_token` The storage account SAS token. This field is ignored if `storage_connection_string` or `storage_access_key` are set. **Type**: `string` **Default**: `""` ### [](#table_name)`table_name` The table to store messages into. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: table_name: ${! meta("kafka_topic") } # --- table_name: ${! json("table") } ``` ### [](#timeout)`timeout` The maximum period to wait on an upload before abandoning it and reattempting. **Type**: `string` **Default**: `5s` ### [](#transaction_type)`transaction_type` Type of transaction operation. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `INSERT` **Options**: `INSERT`, `INSERT_MERGE`, `INSERT_REPLACE`, `UPDATE_MERGE`, `UPDATE_REPLACE`, `DELETE` ```yaml # Examples: transaction_type: ${! json("operation") } # --- transaction_type: ${! meta("operation") } # --- transaction_type: INSERT ``` --- # Page 135: beanstalkd **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/beanstalkd.md --- # beanstalkd --- title: beanstalkd latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/beanstalkd page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/beanstalkd.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/beanstalkd.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/beanstalkd/)[Input](/redpanda-connect/components/inputs/beanstalkd/) **Available in:** Self-Managed Write messages to a Beanstalkd queue. Introduced in version 4.7.0. #### Common ```yml outputs: label: "" beanstalkd: address: "" # No default (required) max_in_flight: 64 ``` #### Advanced ```yml outputs: label: "" beanstalkd: address: "" # No default (required) max_in_flight: 64 ``` ## [](#fields)Fields ### [](#address)`address` An address to connect to. **Type**: `string` ```yaml # Examples: address: 127.0.0.1:11300 ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase to improve throughput. **Type**: `int` **Default**: `64` --- # Page 136: broker **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/broker.md --- # broker --- title: broker latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/broker page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/broker.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/broker.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-03-19" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/broker/)[Input](/redpanda-connect/components/inputs/broker/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/broker/ "View the Cloud version of this component"), Self-Managed A meta-output that routes messages to child outputs using a range of brokering [patterns](#patterns). Unlike regular outputs, `broker` doesn’t send messages anywhere by itself. Instead, it wraps other outputs and controls how messages are delivered across them. Use `broker` to fan out the same message to multiple destinations (for example, publishing events to Kafka while also writing them to a database), or to distribute messages across a pool of outputs for load balancing or throughput scaling. The delivery pattern determines whether each message is written to all outputs or routed to a single output, and whether writes happen in parallel or in sequence. > 📝 **NOTE** > > The name `broker` refers to the brokering delivery pattern, not a Redpanda broker (cluster node). #### Common ```yml outputs: label: "" broker: pattern: fan_out outputs: [] # No default (required) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" broker: copies: 1 pattern: fan_out outputs: [] # No default (required) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` [Processors](../../processors/about/) can be listed to apply across individual outputs or all outputs: ```yaml output: broker: pattern: fan_out outputs: - resource: foo - resource: bar # Processors only applied to messages sent to bar. processors: - resource: bar_processor # Processors applied to messages sent to all brokered outputs. processors: - resource: general_processor ``` ## [](#fields)Fields ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#copies)`copies` The number of copies of each configured output to spawn. **Type**: `int` **Default**: `1` ### [](#outputs)`outputs[]` A list of child outputs to broker. **Type**: `output` ### [](#pattern)`pattern` The brokering pattern to use. **Type**: `string` **Default**: `fan_out` **Options**: `fan_out`, `fan_out_fail_fast`, `fan_out_sequential`, `fan_out_sequential_fail_fast`, `round_robin`, `greedy` ## [](#patterns)Patterns The broker pattern determines how messages are distributed across outputs. Use `fan_out` (the default) when every output should receive every message. Use `round_robin` or `greedy` when you want to distribute messages across outputs for load balancing rather than duplication. The available patterns are: ### [](#fan_out)`fan_out` With the fan out pattern all outputs will be sent every message that passes through Redpanda Connect in parallel. If an output applies back pressure it will block all subsequent messages, and if an output fails to send a message it will be retried continuously until completion or service shut down. This mechanism is in place in order to prevent one bad output from causing a larger retry loop that results in a good output from receiving unbounded message duplicates. Sometimes it is useful to disable the back pressure or retries of certain fan out outputs and instead drop messages that have failed or were blocked. In this case you can wrap outputs with a [`drop_on` output](../drop_on/). ### [](#fan_out_fail_fast)`fan_out_fail_fast` The same as the `fan_out` pattern, except that output failures will not be automatically retried. This pattern should be used with caution as busy retry loops could result in unlimited duplicates being introduced into the non-failure outputs. ### [](#fan_out_sequential)`fan_out_sequential` Similar to the fan out pattern except outputs are written to sequentially, meaning an output is only written to once the preceding output has confirmed receipt of the same message. If an output applies back pressure it will block all subsequent messages, and if an output fails to send a message it will be retried continuously until completion or service shut down. This mechanism is in place in order to prevent one bad output from causing a larger retry loop that results in a good output from receiving unbounded message duplicates. ### [](#fan_out_sequential_fail_fast)`fan_out_sequential_fail_fast` The same as the `fan_out_sequential` pattern, except that output failures will not be automatically retried. This pattern should be used with caution as busy retry loops could result in unlimited duplicates being introduced into the non-failure outputs. ### [](#round_robin)`round_robin` With the round robin pattern each message will be assigned a single output following their order. If an output applies back pressure it will block all subsequent messages. If an output fails to send a message then the message will be re-attempted with the next input, and so on. ### [](#greedy)`greedy` The greedy pattern results in higher output throughput at the cost of potentially disproportionate message allocations to those outputs. Each message is sent to a single output, which is determined by allowing outputs to claim messages as soon as they are able to process them. This results in certain faster outputs potentially processing more messages at the cost of slower outputs. --- # Page 137: cache **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/cache.md --- # cache --- title: cache latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/cache page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/cache.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/cache.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/cache/)[Processor](/redpanda-connect/components/processors/cache/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/cache/ "View the Cloud version of this component"), Self-Managed Stores each message in a [cache](../../caches/about/). #### Common ```yml outputs: label: "" cache: target: "" # No default (required) key: ${!count("items")}-${!timestamp_unix_nano()} max_in_flight: 64 ``` #### Advanced ```yml outputs: label: "" cache: target: "" # No default (required) key: ${!count("items")}-${!timestamp_unix_nano()} ttl: "" # No default (optional) max_in_flight: 64 ``` Caches are configured as [resources](../../caches/about/), where there’s a wide variety to choose from. The `target` field must reference a configured cache resource label like follows: ```yaml output: cache: target: foo key: ${!json("document.id")} cache_resources: - label: foo memcached: addresses: - localhost:11211 default_ttl: 60s ``` In order to create a unique `key` value per item you should use function interpolations described in [Bloblang queries](../../../configuration/interpolation/#bloblang-queries). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ## [](#fields)Fields ### [](#key)`key` The key to store messages by, function interpolation should be used in order to derive a unique key for each message. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `${!count("items")}-${!timestamp_unix_nano()}` ```yaml # Examples: key: ${!count("items")}-${!timestamp_unix_nano()} # --- key: ${!json("doc.id")} # --- key: ${!meta("kafka_key")} ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#target)`target` The target cache to store messages in. **Type**: `string` ### [](#ttl)`ttl` The TTL of each individual item as a duration string. After this period an item will be eligible for removal during the next compaction. Not all caches support per-key TTLs, and those that do not will fall back to their generally configured TTL setting. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). Requires version 3.33.0 or later. **Type**: `string` ```yaml # Examples: ttl: 60s # --- ttl: 5m # --- ttl: 36h ``` --- # Page 138: cassandra **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/cassandra.md --- # cassandra --- title: cassandra latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/cassandra page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/cassandra.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/cassandra.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/cassandra/)[Input](/redpanda-connect/components/inputs/cassandra/) **Available in:** Self-Managed Runs a query against a Cassandra database for each message in order to insert data. #### Common ```yml outputs: label: "" cassandra: addresses: [] # No default (required) timeout: 600ms reconnect_interval: 60s query: "" # No default (required) args_mapping: "" # No default (optional) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" cassandra: addresses: [] # No default (required) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] password_authenticator: enabled: false username: "" password: "" disable_initial_host_lookup: false max_retries: 3 backoff: initial_interval: 1s max_interval: 5s timeout: 600ms host_selection_policy: local_dc: "" # No default (optional) local_rack: "" # No default (optional) reconnect_interval: 60s exponential_reconnection: max_retries: "" # No default (required) initial_interval: "" # No default (required) max_interval: "" # No default (required) query: "" # No default (required) args_mapping: "" # No default (optional) consistency: QUORUM logged_batch: true max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` Query arguments can be set using a bloblang array for the fields using the `args_mapping` field. When populating timestamp columns the value must either be a string in ISO 8601 format (2006-01-02T15:04:05Z07:00), or an integer representing unix time in seconds. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](../../../configuration/batching/). ## [](#examples)Examples ### [](#basic-inserts)Basic Inserts If we were to create a table with some basic columns with `CREATE TABLE foo.bar (id int primary key, content text, created_at timestamp);`, and were processing JSON documents of the form `{"id":"342354354","content":"hello world","timestamp":1605219406}` using logged batches, we could populate our table with the following config: ```yaml output: cassandra: addresses: - localhost:9042 query: 'INSERT INTO foo.bar (id, content, created_at) VALUES (?, ?, ?)' args_mapping: | root = [ this.id, this.content, this.timestamp ] batching: count: 500 period: 1s ``` ### [](#insert-json-documents)Insert JSON Documents The following example inserts JSON documents into the table `footable` of the keyspace `foospace` using INSERT JSON ([https://cassandra.apache.org/doc/latest/cql/json.html#insert-json](https://cassandra.apache.org/doc/latest/cql/json.html#insert-json)). ```yaml output: cassandra: addresses: - localhost:9042 query: 'INSERT INTO foospace.footable JSON ?' args_mapping: 'root = [ this ]' batching: count: 500 period: 1s ``` ## [](#fields)Fields ### [](#addresses)`addresses[]` A list of Cassandra nodes to connect to. Multiple comma separated addresses can be specified on a single line. **Type**: `array` ```yaml # Examples: addresses: - "localhost:9042" # --- addresses: - "foo:9042" - "bar:9042" # --- addresses: - "foo:9042,bar:9042" ``` ### [](#args_mapping)`args_mapping` A [Bloblang mapping](../../../guides/bloblang/about/) that can be used to provide arguments to Cassandra queries. The result of the query must be an array containing a matching number of elements to the query arguments. Requires version 3.55.0 or later. **Type**: `string` ### [](#backoff)`backoff` Control time intervals between retry attempts. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` The initial period to wait between retry attempts. The retry interval increases for each failed attempt, up to the `backoff.max_interval` value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `1s` ### [](#backoff-max_interval)`backoff.max_interval` The maximum period to wait between retry attempts. **Type**: `string` **Default**: `5s` ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#consistency)`consistency` The consistency level to use. **Type**: `string` **Default**: `QUORUM` **Options**: `ANY`, `ONE`, `TWO`, `THREE`, `QUORUM`, `ALL`, `LOCAL_QUORUM`, `EACH_QUORUM`, `LOCAL_ONE` ### [](#disable_initial_host_lookup)`disable_initial_host_lookup` If enabled the driver will not attempt to get host info from the system.peers table. This can speed up queries but will mean that data\_centre, rack and token information will not be available. **Type**: `bool` **Default**: `false` ### [](#exponential_reconnection)`exponential_reconnection` Configure exponential backoff for reconnection attempts to DOWN nodes. When enabled, this replaces the driver’s default constant reconnection policy with an exponential backoff strategy that gradually increases the delay between reconnection attempts. This reduces connection storm scenarios during widespread outages while ensuring eventual recovery. Requires version 4.66.0 or later. **Type**: `object` ### [](#exponential_reconnection-initial_interval)`exponential_reconnection.initial_interval` The initial period to wait between retry attempts. **Type**: `string` ### [](#exponential_reconnection-max_interval)`exponential_reconnection.max_interval` The maximum period to wait between retry attempts. **Type**: `string` ### [](#exponential_reconnection-max_retries)`exponential_reconnection.max_retries` The maximum number of retry attempts. **Type**: `int` ### [](#host_selection_policy)`host_selection_policy` Advanced host selection policy settings for Cassandra clusters. Use these options to optimize query routing in multi-datacenter (DC) and multi-rack deployments. By specifying a local DC and rack, you can ensure queries are directed to the closest nodes, reducing latency and improving fault tolerance. If not set, the default policy is round-robin across all available nodes. Host selection is always token-aware if the token can be calculated from query. Requires version 4.61.0 or later. **Type**: `object` ```yaml # Examples: host_selection_policy: local_dc: dc-east local_rack: rack1 ``` ### [](#host_selection_policy-local_dc)`host_selection_policy.local_dc` The name of the local datacenter to prioritize for query routing. Enables DC-aware host selection, ensuring queries are sent to nodes within this datacenter whenever possible. Recommended for clusters spanning multiple datacenters to minimize cross-DC traffic. **Type**: `string` ### [](#host_selection_policy-local_rack)`host_selection_policy.local_rack` The name of the local rack to prioritize for query routing. Requires `local_dc` to be set. Enables rack-aware host selection, further optimizing query placement within the specified datacenter. Useful for deployments with multiple racks per datacenter to improve resilience and reduce intra-DC latency. **Type**: `string` ### [](#logged_batch)`logged_batch` If enabled the driver will perform a logged batch. Disabling this prompts unlogged batches to be used instead, which are less efficient but necessary for alternative storages that do not support logged batches. **Type**: `bool` **Default**: `true` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#max_retries)`max_retries` The maximum number of retries before giving up on a request. **Type**: `int` **Default**: `3` ### [](#password_authenticator)`password_authenticator` Optional configuration of Cassandra authentication parameters. **Type**: `object` ### [](#password_authenticator-enabled)`password_authenticator.enabled` Whether to use password authentication **Type**: `bool` **Default**: `false` ### [](#password_authenticator-password)`password_authenticator.password` The password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#password_authenticator-username)`password_authenticator.username` The username to authenticate as. **Type**: `string` **Default**: `""` ### [](#query)`query` A query to execute for each message. **Type**: `string` ### [](#reconnect_interval)`reconnect_interval` The interval at which Redpanda Connect attempts to reconnect to Cassandra nodes that are marked as DOWN. This setting helps maintain connectivity in unstable network conditions or during node maintenance. Use Go duration format such as `30s`, `1m`, or `5m`. Setting this too low may create unnecessary connection attempts, while setting it too high may delay recovery from network issues. Requires version 4.66.0 or later. **Type**: `string` **Default**: `60s` ### [](#timeout)`timeout` The client connection timeout. **Type**: `string` **Default**: `600ms` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` --- # Page 139: couchbase **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/couchbase.md --- # couchbase --- title: couchbase latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/couchbase page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/couchbase.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/couchbase.adoc page-git-created-date: "2024-10-16" page-git-modified-date: "2026-01-05" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/couchbase/)[Cache](/redpanda-connect/components/caches/couchbase/)[Processor](/redpanda-connect/components/processors/couchbase/) **Available in:** Self-Managed Perform operations on a [Couchbase database](https://docs.couchbase.com/home/index.html) for each incoming message. This connector can store or delete data in Couchbase based on the message’s content. Introduced in version 4.37.0. #### Common ```yml outputs: label: "" couchbase: url: "" # No default (required) username: "" # No default (optional) password: "" # No default (optional) bucket: "" # No default (required) id: "" # No default (required) content: "" # No default (optional) operation: upsert max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" couchbase: url: "" # No default (required) username: "" # No default (optional) password: "" # No default (optional) bucket: "" # No default (required) collection: "" # No default (optional) scope: "" # No default (optional) transcoder: legacy timeout: 15s id: "" # No default (required) content: "" # No default (optional) ttl: "" # No default (optional) operation: upsert max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#performance)Performance For improved performance, this output sends multiple messages in parallel. You can tune the maximum number of in-flight messages (or message batches), using the `max_in_flight` field. You can configure batches at both the input and output level. For more information, see [Message Batching](../../../configuration/batching/). ## [](#fields)Fields ### [](#batching)`batching` Configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#bucket)`bucket` The name of the Couchbase bucket you want the output to access. **Type**: `string` ### [](#collection)`collection` The name of the specific collection you want the output to write to. **Type**: `string` ### [](#content)`content` The document content to update. When inserting, replacing, or upserting documents, you must set a `content` value. **Type**: `string` ### [](#id)`id` The document ID to use. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: id: ${! json("id") } ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#operation)`operation` The Couchbase operation to perform. **Type**: `string` **Default**: `upsert` | Option | Summary | | --- | --- | | insert | insert a new document. | | remove | delete a document. | | replace | replace the contents of a document. | | upsert | creates a new document if it does not exist, if it does exist then it updates it. | ### [](#password)`password` A password to authenticate with the Couchbase cluster. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#scope)`scope` Bucket scope. **Type**: `string` ### [](#timeout)`timeout` Operation timeout if this output cannot connect to the Couchbase cluster. **Type**: `string` **Default**: `15s` ### [](#transcoder)`transcoder` Choose from the following Couchbase transcoders to convert the messages' data format before they are written to Couchbase. **Type**: `string` **Default**: `legacy` | Option | Summary | | --- | --- | | json | JSONTranscoder implements the default transcoding behavior and applies JSON transcoding to all values. This will apply the following behavior to the value: binary ([]byte) → error. default → JSON value, JSON Flags. | | legacy | LegacyTranscoder implements the behavior for a backward-compatible transcoder. This transcoder implements behavior matching that of gocb v1.This will apply the following behavior to the value: binary ([]byte) → binary bytes, Binary expectedFlags. string → string bytes, String expectedFlags. default → JSON value, JSON expectedFlags. | | raw | RawBinaryTranscoder implements passthrough behavior of raw binary data. This transcoder does not apply any serialization. This will apply the following behavior to the value: binary ([]byte) → binary bytes, binary expectedFlags. default → error. | | rawjson | RawJSONTranscoder implements passthrough behavior of JSON data. This transcoder does not apply any serialization. It will forward data across the network without incurring unnecessary parsing costs. This will apply the following behavior to the value: binary ([]byte) → JSON bytes, JSON expectedFlags. string → JSON bytes, JSON expectedFlags. default → error. | | rawstring | RawStringTranscoder implements passthrough behavior of raw string data. This transcoder does not apply any serialization. This will apply the following behavior to the value: string → string bytes, string expectedFlags. default → error. | ### [](#ttl)`ttl` An optional TTL to set for items. **Type**: `string` ### [](#url)`url` The Couchbase connection string. **Type**: `string` ```yaml # Examples: url: couchbase://localhost:11210 ``` ### [](#username)`username` A username to authenticate with the Couchbase cluster. **Type**: `string` --- # Page 140: cyborgdb **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/cyborgdb.md --- # cyborgdb --- title: cyborgdb latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/cyborgdb page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/cyborgdb.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/cyborgdb.adoc description: Inserts items into a CyborgDB encrypted vector index. page-git-created-date: "2025-10-06" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/cyborgdb/ "View the Cloud version of this component"), Self-Managed Inserts items into a CyborgDB encrypted vector index. Introduced in version 4.66.0 #### Common ```yaml outputs: label: "" cyborgdb: max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) host: "" # No default (required) api_key: "" # No default (required) index_name: redpanda-vectors index_key: "" # No default (required) operation: upsert id: "" # No default (required) vector_mapping: "" # No default (optional) metadata_mapping: "" # No default (optional) ``` #### Advanced ```yaml outputs: label: "" cyborgdb: max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) host: "" # No default (required) api_key: "" # No default (required) index_name: redpanda-vectors index_key: "" # No default (required) create_if_missing: false operation: upsert id: "" # No default (required) vector_mapping: "" # No default (optional) metadata_mapping: "" # No default (optional) ``` This output allows you to write vectors to a CyborgDB encrypted index. CyborgDB provides end-to-end encrypted vector storage with automatic dimension detection and index optimization. All vector data is encrypted client-side before being sent to the server, ensuring complete data privacy. The encryption key never leaves your infrastructure. ## [](#fields)Fields ### [](#api_key)`api_key` The API key for authenticating with the CyborgDB service. This key identifies your account and provides access to your CyborgDB indexes. Keep this key secure and avoid exposing it in logs or version control. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#create_if_missing)`create_if_missing` Whether to create the index if it doesn’t exist. When enabled, CyborgDB automatically detects the vector dimensions from your data and optimizes the index configuration for performance. This is useful for development and testing environments. **Type**: `bool` **Default**: `false` ### [](#host)`host` The host URL for the CyborgDB instance. This should include the protocol (https://) and port number if required. For example: `[https://api.cyborgdb.com](https://api.cyborgdb.com)` or `[https://localhost:8080](https://localhost:8080)`. **Type**: `string` ```yaml # Examples: host: api.cyborg.com # --- host: localhost:8000 ``` ### [](#id)`id` A [Bloblang mapping](../../../guides/bloblang/about/) that determines the unique identifier for each vector entry. This ID is used to update existing vectors during upsert operations or to specify which vectors to delete. If not provided, CyborgDB will generate unique IDs automatically. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#index_key)`index_key` The base64-encoded encryption key for the CyborgDB index. This key must be exactly 32 bytes when decoded from base64. All vector data is encrypted client-side using this key before transmission, ensuring complete data privacy. Store this key securely as it cannot be recovered if lost. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ```yaml # Examples: index_key: your-base64-encoded-32-byte-key ``` ### [](#index_name)`index_name` The name of the CyborgDB index to write vectors to. If the index doesn’t exist and `create_if_missing` is enabled, CyborgDB will create it automatically with optimized settings based on your data. **Type**: `string` **Default**: `redpanda-vectors` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#metadata_mapping)`metadata_mapping` An optional [Bloblang mapping](../../../guides/bloblang/about/) that extracts metadata to associate with the vector entry. The metadata can contain any JSON-serializable data that helps identify or categorize the vector. This data is stored encrypted alongside the vector. **Type**: `string` ```yaml # Examples: metadata_mapping: root = @ # --- metadata_mapping: root = metadata() # --- metadata_mapping: root = {"summary": this.summary, "category": this.category} ``` ### [](#operation)`operation` The operation to perform against the CyborgDB index. Supported operations: - `upsert`: Insert new vectors or update existing ones (requires `vector_mapping`) - `delete`: Remove vectors from the index (requires `id`) - `query`: Search for similar vectors (requires `vector_mapping`) **Type**: `string` **Default**: `upsert` **Options**: `upsert`, `delete` ### [](#vector_mapping)`vector_mapping` A [Bloblang mapping](../../../guides/bloblang/about/) that extracts the vector from the message. The result must be an array of floating-point numbers representing the vector embeddings. This field is required for `upsert` and `query` operations. **Type**: `string` ```yaml # Examples: vector_mapping: root = this.embeddings_vector # --- vector_mapping: root = [1.2, 0.5, 0.76] ``` --- # Page 141: cypher **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/cypher.md --- # cypher --- title: cypher latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/cypher page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/cypher.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/cypher.adoc page-git-created-date: "2024-10-16" page-git-modified-date: "2025-10-06" --- **Available in:** Self-Managed Writes a batch of messages to any graph database that supports the [Neo4j and Bolt URI schemes](https://neo4j.com/docs/go-manual/current/connect-advanced/). For each incoming message, the connector can perform operations to store or delete data within the database using the [Cypher query language](https://neo4j.com/docs/cypher-manual/current/introduction/). Introduced in version 4.37.0. #### Common ```yml outputs: label: "" cypher: uri: "" # No default (required) cypher: "" # No default (required) database_name: "" args_mapping: "" # No default (optional) basic_auth: enabled: false username: "" password: "" realm: "" batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_in_flight: 64 ``` #### Advanced ```yml outputs: label: "" cypher: uri: "" # No default (required) cypher: "" # No default (required) database_name: "" args_mapping: "" # No default (optional) basic_auth: enabled: false username: "" password: "" realm: "" tls: skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_in_flight: 64 ``` ## [](#examples)Examples ### [](#write-to-neo4j-aura)Write to Neo4j Aura This is an example of how to write to Neo4j Aura ```yaml output: cypher: uri: neo4j+s://example.databases.neo4j.io cypher: | MERGE (product:Product {id: $id}) ON CREATE SET product.name = $product, product.title = $title, product.description = $description, args_mapping: | root = {} root.id = this.product.id root.product = this.product.summary.name root.title = this.product.summary.displayName root.description = this.product.fullDescription basic_auth: enabled: true username: "${NEO4J_USER}" password: "${NEO4J_PASSWORD}" ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` Mappings from incoming messages to the data, which are passed into the cypher expression as parameters. All mappings must be objects. By default, this field processes the entire payload. **Type**: `string` ```yaml # Examples: args_mapping: root.name = this.displayName # --- args_mapping: root = {"orgId": this.org.id, "name": this.user.name} ``` ### [](#basic_auth)`basic_auth` Configure basic authentication for requests to your graphing database. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` The password to use for authentication. Used together with `username` for basic authentication or with encrypted private keys for secure access. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-realm)`basic_auth.realm` The realm or process for authentication challenges. **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` The username of the account credentials to authenticate as. Used together with `password` for basic authentication. **Type**: `string` **Default**: `""` ### [](#batching)`batching` Configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#cypher)`cypher` The cypher expression to execute against the graph database. **Type**: `string` ```yaml # Examples: cypher: MERGE (p:Person {name: $name}) # --- cypher: |- MATCH (o:Organization {id: $orgId}) MATCH (p:Person {name: $name}) MERGE (p)-[:WORKS_FOR]->(o) ``` ### [](#database_name)`database_name` Set the target database against which expressions are evaluated. **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of message batches to have in flight at a given time. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#uri)`uri` The connection URI for your graphing database. For more information, see [Neo4j’s documentation](https://neo4j.com/docs/go-manual/current/connect-advanced/). **Type**: `string` ```yaml # Examples: uri: neo4j://demo.neo4jlabs.com # --- uri: neo4j+s://aura.databases.neo4j.io # --- uri: neo4j+ssc://self-signed.demo.neo4jlabs.com # --- uri: bolt://127.0.0.1:7687 # --- uri: bolt+s://core.db.server:7687 # --- uri: bolt+ssc://10.0.0.43 ``` --- # Page 142: discord **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/discord.md --- # discord --- title: discord latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/discord page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/discord.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/discord.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/discord/)[Input](/redpanda-connect/components/inputs/discord/) **Available in:** Self-Managed Writes messages to a Discord channel. ```yml outputs: label: "" discord: channel_id: "" # No default (required) bot_token: "" # No default (required) ``` This output POSTs messages to the `/channels/{channel_id}/messages` Discord API endpoint authenticated as a bot using token based authentication. If the format of a message is a JSON object matching the [Discord API message type](https://discord.com/developers/docs/resources/channel#message-object) then it is sent directly, otherwise an object matching the API type is created with the content of the message added as a string. ## [](#fields)Fields ### [](#bot_token)`bot_token` A bot token used for authentication. **Type**: `string` ### [](#channel_id)`channel_id` A discord channel ID to write messages to. **Type**: `string` --- # Page 143: drop_on **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/drop_on.md --- # drop\_on --- title: drop_on latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/drop_on page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/drop_on.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/drop_on.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/drop_on/ "View the Cloud version of this component"), Self-Managed Attempts to write messages to a child output and if the write fails for one of a list of configurable reasons the message is dropped (acked) instead of being reattempted (or nacked). ```yml outputs: label: "" drop_on: error: false error_patterns: [] # No default (optional) back_pressure: "" # No default (optional) output: "" # No default (required) ``` Regular Redpanda Connect outputs will apply back pressure when downstream services aren’t accessible, and Redpanda Connect retries (or nacks) all messages that fail to be delivered. However, in some circumstances, or for certain output types, we instead might want to relax these mechanisms, which is when this output becomes useful. ## [](#fields)Fields ### [](#back_pressure)`back_pressure` An optional duration string that determines the maximum length of time to wait for a given message to be accepted by the child output before the message should be dropped instead. The most common reason for an output to block is when waiting for a lost connection to be re-established. Once a message has been dropped due to back pressure all subsequent messages are dropped immediately until the output is ready to process them again. Note that if `error` is set to `false` and this field is specified then messages dropped due to back pressure will return an error response (are nacked or reattempted). **Type**: `string` ```yaml # Examples: back_pressure: 30s # --- back_pressure: 1m ``` ### [](#error)`error` Whether messages should be dropped when the child output returns an error of any type. For example, this could be when an `http_client` output gets a 4XX response code. In order to instead drop only on specific error patterns use the `error_matches` field instead. **Type**: `bool` **Default**: `false` ### [](#error_patterns)`error_patterns[]` A list of regular expressions (re2) where if the child output returns an error that matches any part of any of these patterns the message will be dropped. Requires version 4.27.0 or later. **Type**: `array` ```yaml # Examples: error_patterns: - "and that was really bad$" # --- error_patterns: - "roughly [0-9]+ issues occurred" ``` ### [](#output)`output` A child output to wrap with this drop mechanism. **Type**: `output` nclude::redpanda-connect:components:partial$examples/outputs/drop\_on.adoc\[\] --- # Page 144: drop **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/drop.md --- # drop --- title: drop latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/drop page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/drop.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/drop.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/drop/ "View the Cloud version of this component"), Self-Managed Drops all messages. ```yml outputs: label: "" drop: {} ``` --- # Page 145: dynamic **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/dynamic.md --- # dynamic --- title: dynamic latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/dynamic page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/dynamic.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/dynamic.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/dynamic/)[Input](/redpanda-connect/components/inputs/dynamic/) **Available in:** Self-Managed A special broker type where the outputs are identified by unique labels and can be created, changed and removed during runtime via a REST API. ```yml outputs: label: "" dynamic: outputs: {} prefix: "" ``` The broker pattern used is always `fan_out`, meaning each message will be delivered to each dynamic output. ## [](#fields)Fields ### [](#outputs)`outputs` A map of outputs to statically create. **Type**: `output` **Default**: `{}` ### [](#prefix)`prefix` A path prefix for HTTP endpoints that are registered. **Type**: `string` **Default**: `""` ## [](#endpoints)Endpoints ### [](#get-outputs)GET `/outputs` Returns a JSON object detailing all dynamic outputs, providing information such as their current uptime and configuration. ### [](#get-outputsid)GET `/outputs/{id}` Returns the configuration of an output. ### [](#post-outputsid)POST `/outputs/{id}` Creates or updates an output with a configuration provided in the request body (in YAML or JSON format). ### [](#delete-outputsid)DELETE `/outputs/{id}` Stops and removes an output. ### [](#get-outputsiduptime)GET `/outputs/{id}/uptime` Returns the uptime of an output as a duration string (of the form "72h3m0.5s"). --- # Page 146: elasticsearch_v8 **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/elasticsearch_v8.md --- # elasticsearch\_v8 --- title: elasticsearch_v8 latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/elasticsearch_v8 page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/elasticsearch_v8.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/elasticsearch_v8.adoc page-git-created-date: "2025-03-12" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/elasticsearch_v8/ "View the Cloud version of this component"), Self-Managed Publishes messages into an [Elasticsearch index](https://www.elastic.co/guide/en/elasticsearch/reference/current/documents-indices.html). If the index does not exist, this output creates it using dynamic mapping. > 📝 **NOTE** > > The `elasticsearch_v8` output is based on the the [go-elasticsearch/v8](https://github.com/elastic/go-elasticsearch?tab=readme-ov-file) library. For full information about breaking changes from previous versions, see [Elastic’s Migrating to 8.0 guide](https://www.elastic.co/guide/en/elasticsearch/reference/current/migrating-8.0.html#breaking_80_rest_api_changes). To help configure your own `elasticsearch_v8` output, this page includes [example pipeline configurations](#example-pipelines). Introduced in version 4.47.0. #### Common ```yml outputs: label: "" elasticsearch_v8: urls: [] # No default (required) index: "" # No default (required) action: "" # No default (required) id: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" elasticsearch_v8: urls: [] # No default (required) index: "" # No default (required) action: "" # No default (required) id: "" # No default (required) pipeline: "" routing: "" retry_on_conflict: 0 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] max_in_flight: 64 basic_auth: enabled: false username: "" password: "" batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#set-values-dynamically)Set values dynamically You can use [function interpolations](../../../configuration/interpolation/#bloblang-queries) to dynamically set values for the [`id`](#id) and [`index`](#index) fields, as well as other fields where [function interpolations](../../../configuration/interpolation/#bloblang-queries) are supported. When message batches are sent, interpolations are performed per message. ## [](#performance)Performance For improved performance, this output sends: - Multiple messages in parallel. Adjust the `max_in_flight` field value to tune the maximum number of in-flight messages (or message batches). - Messages as batches. You can configure batches at both input and output level. For more information, see [Message Batching](../../../configuration/batching/). ## [](#fields)Fields ### [](#action)`action` The action to perform on each document. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). For more information on how the `update` action works, see [Example pipelines](#example-pipelines). **Type**: `string` ### [](#basic_auth)`basic_auth` Configure basic authentication credentials for connecting to Elasticsearch. When enabled, these credentials are sent with each request to authenticate with the cluster. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#batching)`batching` Configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#id)`id` Define the ID for indexed messages. Use [function interpolations](../../../configuration/interpolation/#bloblang-queries) to dynamically create a unique ID for each message. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: id: ${!counter()}-${!timestamp_unix()} ``` ### [](#index)`index` The Elasticsearch index where messages are published. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#pipeline)`pipeline` Specify the ID of a pipeline to preprocess incoming documents before they are published (optional). This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#retry_on_conflict)`retry_on_conflict` The number of times to retry an update operation when a version conflict occurs. **Type**: `int` **Default**: `0` ### [](#routing)`routing` The routing key to use for the document. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. This output attempts to connect to each URL in the list, in order, until a successful connection is established. If an item in the list contains commas, it is split into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "http://localhost:9200" ``` ## [](#example-pipelines)Example pipelines ### Update documents To update documents in the target index, the top level of the request body must include at least one of the following fields: - `doc`: Performs partial updates on a document. - `upsert`: Updates an existing document or inserts a document if it doesn’t exist. - `script`: Performs an update using a scripting language, such as [Elasticsearch’s Painless scripting language](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting-painless.html). The following examples show how to configure mapping processors with this output to achieve different types of updates. Example 1: Partial document update ```yaml output: processors: # Sets the metadata ID field to the message ID then # performs a partial update on the document. - mapping: | meta id = this.id root.doc = this elasticsearch_v8: urls: [localhost:9200] # The URL of the Elasticsearch server. index: my_target_index # The name of the Elasticsearch index. id: ${! @id } # Sets the document ID to the value of the metadata ID field. action: update # The action to perform on each document. ``` Example 2: Scripted update ```yaml output: processors: # Sets the metadata ID field to the message ID then # increments the counter field by `1` using a script. - mapping: | meta id = this.id root.script.source = "ctx._source.counter += 1" elasticsearch_v8: urls: [localhost:9200] # The URL of the Elasticsearch server. index: my_target_index # The name of the Elasticsearch index. id: ${! @id } # Sets the document ID to the value of the metadata ID field. action: update # The action to perform on each document. ``` Example 3: Upsert ```yaml output: processors: # Sets the metadata ID field to the message ID. # If the product with the specified ID exists, update its product_price to 100. # If the document does not exist, insert a new document with the ID set to 1 # and the `product_price` set to 50. - mapping: | meta id = this.id root.doc.product_price = 100 root.upsert.product_price = 50 elasticsearch_v8: urls: [localhost:9200] # The URL of the Elasticsearch server. index: my_target_index # The name of the Elasticsearch index. id: ${! @id } # Sets the document ID to the value of the metadata ID field. action: update # The action to perform on each document. ``` For more information on the structures and behaviors of `doc`, `upsert`, and `script` fields, see the [Elasticsearch Update API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html). ### Index documents from Redpanda Reads messages from a Redpanda cluster and writes them to an Elasticsearch index using a field from the message as the document ID. ```yaml # Reads messages from a Redpanda cluster. input: redpanda: seed_brokers: [localhost:19092] # The address of the Redpanda broker. topics: ["product_code"] # The topic to consume messages from. consumer_group: "rpcn3" # The consumer group ID. processors: # Sets the metadata ID field to the message ID and # sets the root of the message to the message content. - mapping: | meta id = this.id root = this # Writes messages to the specified Elasticsearch index. output: elasticsearch_v8: urls: ['http://localhost:9200'] # The URL of the Elasticsearch server. index: "product_code" # The name of the Elasticsearch index. action: "index" # The action to perform on each document. id: ${! meta("id") } # Sets the document ID to the value of the metadata ID field. ``` ### Index documents from AWS S3 Reads messages from a AWS S3 bucket and writes them to an Elasticsearch index using the S3 key as the ID for the Elasticsearch document. ```yaml # Reads messages from an AWS S3 bucket. input: aws_s3: bucket: "my_bucket" # The name of the S3 bucket. prefix: "prod_inventory/" # A prefix to filter objects in the bucket. scanner: to_the_end: {} # Scans the bucket to the end. # Writes messages to the specified Elasticsearch index. output: elasticsearch_v8: urls: ['http://localhost:9200'] # The URL of the Elasticsearch server. index: "current_prod_inventory" # The name of the Elasticsearch index. action: "index" # The action to perform on each document. id: ${! meta("s3_key") } # Sets the document ID to the S3 key. ``` --- # Page 147: elasticsearch_v9 **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/elasticsearch_v9.md --- # elasticsearch\_v9 --- title: elasticsearch_v9 latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/elasticsearch_v9 page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/elasticsearch_v9.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/elasticsearch_v9.adoc description: Publishes messages into an Elasticsearch index. If the index does not exist then it is created with a dynamic mapping. page-git-created-date: "2026-01-19" page-git-modified-date: "2026-01-19" --- **Available in:** Self-Managed Publishes messages into an Elasticsearch index. If the index does not exist, the output creates one with a dynamic mapping. Both the `id` and `index` fields can be dynamically set using [function interpolations](../../../configuration/interpolation/#bloblang-queries). When sending batched messages these interpolations are performed per message part. Introduced in version 4.77.0. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in-flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. You can compose a batch at both the input and output level. For more information, see [Message Batching](../../../configuration/batching/). ### Common ```yml outputs: label: "" elasticsearch_v9: urls: [] # No default (required) index: "" # No default (required) action: "" # No default (required) id: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ### Advanced ```yml outputs: label: "" elasticsearch_v9: urls: [] # No default (required) index: "" # No default (required) action: "" # No default (required) id: "" # No default (required) pipeline: "" routing: "" retry_on_conflict: 0 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] max_in_flight: 64 basic_auth: enabled: false username: "" password: "" batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#fields)Fields ### [](#action)`action` The action to perform on each document. This field must resolve to one of the following action types: `index`, `update`, `delete`, `create` or `upsert`. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). For more information on how the `update` action works, see [Updating Documents](#updating-documents). For examples of how to use the `create` and `upsert` actions, see [Create Documents](#create-documents) and [Upserting Documents](#upserting-documents) respectively. **Type**: `string` ### [](#basic_auth)`basic_auth` Configure basic authentication credentials for connecting to Elasticsearch. When enabled, these credentials are sent with each request to authenticate with the cluster. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#batching)`batching` Configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#id)`id` Define the ID for indexed messages. Use [function interpolations](../../../configuration/interpolation/#bloblang-queries) to dynamically create a unique ID for each message. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: id: ${!counter()}-${!timestamp_unix()} ``` ### [](#index)`index` The Elasticsearch index where messages are published. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#pipeline)`pipeline` Specify the ID of a pipeline to preprocess incoming documents before they are published (optional). This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#retry_on_conflict)`retry_on_conflict` The number of times to retry an update operation when a version conflict occurs. **Type**: `int` **Default**: `0` ### [](#routing)`routing` The routing key to use for the document. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. This output attempts to connect to each URL in the list, in order, until a successful connection is established. If an item in the list contains commas, it is split into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "http://localhost:9200" ``` ## [](#examples)Examples ### [](#updating-documents)Updating Documents When updating documents, the request body should contain a combination of a `doc`, `upsert`, and/or `script` fields at the top level, this should be done via mapping processors. `doc` updates using a partial document, `script` performs an update using a scripting language such as the built in Painless language, and `upsert` updates an existing document or inserts a new one if it doesn’t exist. For more information on the structures and behaviors of these fields, please see the [Elasticsearch Update API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html) ```yaml # Partial document update output: processors: - mapping: | meta id = this.id # Performs a partial update on the document. root.doc = this elasticsearch_v9: urls: [localhost:9200] index: foo id: ${! @id } action: update # Scripted update output: processors: - mapping: | meta id = this.id # Increments the field "counter" by 1. root.script.source = "ctx._source.counter += 1" elasticsearch_v9: urls: [localhost:9200] index: foo id: ${! @id } action: update # Upsert output: processors: - mapping: | meta id = this.id # If the product with the ID exists, its price will be updated to 50. # If the product does not exist, a new document with ID 1 and a price # of 100 will be inserted. root.doc.product_price = 50 root.upsert.product_price = 100 elasticsearch_v9: urls: [localhost:9200] index: foo id: ${! @id } action: update ``` ### [](#indexing-documents-from-redpanda)Indexing documents from Redpanda Here we read messages from a Redpanda cluster and write them to an Elasticsearch index using a field from the message as the ID for the Elasticsearch document. ```yaml input: redpanda: seed_brokers: [localhost:19092] topics: ["things"] consumer_group: "rpcn3" processors: - mapping: | meta id = this.id root = this output: elasticsearch_v9: urls: ['http://localhost:9200'] index: "things" action: "index" id: ${! meta("id") } ``` ### [](#indexing-documents-from-s3)Indexing documents from S3 Here we read messages from a AWS S3 bucket and write them to an Elasticsearch index using the S3 key as the ID for the Elasticsearch document. ```yaml input: aws_s3: bucket: "my-cool-bucket" prefix: "bug-facts/" scanner: to_the_end: {} output: elasticsearch_v9: urls: ['http://localhost:9200'] index: "cool-bug-facts" action: "index" id: ${! meta("s3_key") } ``` ### [](#create-documents)Create Documents When using the `create` action, a new document will be created if the document ID does not already exist. If the document ID already exists, the operation will fail. ```yaml output: elasticsearch_v9: urls: [localhost:9200] index: foo id: ${! json("id") } action: create ``` ### [](#upserting-documents)Upserting Documents When using the `upsert` action, if the document ID already exists, it will be updated. If the document ID does not exist, a new document will be inserted. The request body should contain the document to be indexed. ```yaml output: processors: - mapping: | meta id = this.id root = this.doc elasticsearch_v9: urls: [localhost:9200] index: foo id: ${! @id } action: upsert ``` --- # Page 148: fallback **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/fallback.md --- # fallback --- title: fallback latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/fallback page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/fallback.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/fallback.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/fallback/ "View the Cloud version of this component"), Self-Managed Attempts to send each message to a child output, starting from the first output on the list. If an output attempt fails then the next output in the list is attempted, and so on. Introduced in version 3.58.0. ```yml outputs: label: "" fallback: - label: "" stdout: codec: lines - label: "" file: path: /tmp/fallback.txt codec: lines ``` This pattern is useful for triggering events in the case where certain output targets have broken. For example, if you had an output type `http_client` but wished to reroute messages whenever the endpoint becomes unreachable you could use this pattern: ```yaml output: fallback: - http_client: url: http://foo:4195/post/might/become/unreachable retries: 3 retry_period: 1s - http_client: url: http://bar:4196/somewhere/else retries: 3 retry_period: 1s processors: - mapping: 'root = "failed to send this message to foo: " + content()' - file: path: /usr/local/benthos/everything_failed.jsonl ``` ## [](#metadata)Metadata When a given output fails the message routed to the following output will have a metadata value named `fallback_error` containing a string error message outlining the cause of the failure. The content of this string will depend on the particular output and can be used to enrich the message or provide information used to broker the data to an appropriate output using something like a `switch` output. ## [](#batching)Batching When an output within a fallback sequence uses batching, like so: ```yaml output: fallback: - aws_dynamodb: table: foo string_columns: id: ${!json("id")} content: ${!content()} batching: count: 10 period: 1s - file: path: /usr/local/benthos/failed_stuff.jsonl ``` Redpanda Connect makes a best attempt at inferring which specific messages of the batch failed, and only propagates those individual messages to the next fallback tier. However, depending on the output and the error returned it is sometimes not possible to determine the individual messages that failed, in which case the whole batch is passed to the next tier in order to preserve at-least-once delivery guarantees. --- # Page 149: file **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/file.md --- # file --- title: file latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/file page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/file.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/file.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/file/)[Cache](/redpanda-connect/components/caches/file/)[Input](/redpanda-connect/components/inputs/file/) **Available in:** Self-Managed Writes messages to files on disk based on a chosen codec. #### Common ```yml outputs: label: "" file: path: "" # No default (required) codec: lines ``` #### Advanced ```yml outputs: label: "" file: path: "" # No default (required) codec: lines ``` Messages can be written to different files by using [interpolation functions](../../../configuration/interpolation/#bloblang-queries) in the path field. However, only one file is ever open at a given time, and therefore when the path changes the previously open file is closed. ## [](#fields)Fields ### [](#codec)`codec` The way in which the bytes of messages should be written out into the output data stream. It’s possible to write lines using a custom delimiter with the `delim:x` codec, where x is the character sequence custom delimiter. Requires version 3.33.0 or later. **Type**: `string` **Default**: `lines` | Option | Summary | | --- | --- | | all-bytes | Only applicable to file based outputs. Writes each message to a file in full, if the file already exists the old content is deleted. | | append | Append each message to the output stream without any delimiter or special encoding. | | lines | Append each message to the output stream followed by a line break. | | delim:x | Append each message to the output stream followed by a custom delimiter. | ```yaml # Examples: codec: lines # --- codec: delim: # --- codec: delim:foobar ``` ### [](#path)`path` The file to write to, if the file does not yet exist it will be created. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). Requires version 3.33.0 or later. **Type**: `string` ```yaml # Examples: path: /tmp/data.txt # --- path: /tmp/${! timestamp_unix() }.txt # --- path: /tmp/${! json("document.id") }.json ``` --- # Page 150: gcp_bigquery **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/gcp_bigquery.md --- # gcp\_bigquery --- title: gcp_bigquery latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/gcp_bigquery page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/gcp_bigquery.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/gcp_bigquery.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/gcp_bigquery/ "View the Cloud version of this component"), Self-Managed Inserts message data as new rows in a Google Cloud BigQuery table. Introduced in version 3.55.0. #### Common ```yml outputs: label: "" gcp_bigquery: project: "" job_project: "" dataset: "" # No default (required) table: "" # No default (required) format: NEWLINE_DELIMITED_JSON max_in_flight: 64 job_labels: {} credentials_json: "" csv: header: [] field_delimiter: , allow_jagged_rows: false allow_quoted_newlines: false encoding: UTF-8 skip_leading_rows: 1 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" gcp_bigquery: project: "" job_project: "" dataset: "" # No default (required) table: "" # No default (required) format: NEWLINE_DELIMITED_JSON max_in_flight: 64 write_disposition: WRITE_APPEND create_disposition: CREATE_IF_NEEDED ignore_unknown_values: false max_bad_records: 0 auto_detect: false job_labels: {} credentials_json: "" csv: header: [] field_delimiter: , allow_jagged_rows: false allow_quoted_newlines: false encoding: UTF-8 skip_leading_rows: 1 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#credentials)Credentials By default, Redpanda Connect uses a [shared credentials file](../../../guides/cloud/gcp/) when connecting to GCP services. ## [](#format)Format The `gcp_bigquery` output currently supports only `NEWLINE_DELIMITED_JSON`, `CSV` and `PARQUET` formats. To learn more about how to use BigQuery with these formats, see the following documentation: - [`NEWLINE_DELIMITED_JSON`](https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-json) - [`CSV`](https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-csv) - [`PARQUET`](https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-parquet) ### [](#newline-delimited-json)Newline-delimited JSON Each JSON message may contain multiple elements separated by newlines. For example, a single message containing: ```json {"key": "1"} {"key": "2"} ``` Is equivalent to two separate messages: ```json {"key": "1"} ``` And: ```json {"key": "2"} ``` The same is true for the CSV format. ### [](#csv)CSV When the field `csv.header` is specified for the `CSV` format, a header row is inserted as the first line of each message batch. If this field is not provided, then the first message of each message batch must include a header line. ### [](#parquet)Parquet Each message sent to this output must be a Parquet file. You can use the [`parquet_encode` processor](../../processors/parquet_encode/) to convert message data into the correct format. For example: ```yaml input: generate: mapping: | root = { "foo": random_int(), "bar": uuid_v4(), "time": now(), } interval: 0 count: 1000 batch_size: 1000 pipeline: processors: - parquet_encode: schema: - name: foo type: INT64 - name: bar type: UTF8 - name: time type: UTF8 default_compression: zstd output: gcp_bigquery: project: "${PROJECT}" dataset: "my_bq_dataset" table: "redpanda_connect_ingest" format: PARQUET ``` ## [](#performance)Performance The `gcp_bigquery` output benefits from sending multiple messages in parallel for improved performance. You can tune the maximum number of in-flight messages (or message batches) with the field `max_in_flight`. This output also sends messages as a batch for improved performance. Redpanda Connect can form batches at both the input and output level. For more information, see [Message Batching](../../../configuration/batching/). ## [](#fields)Fields ### [](#auto_detect)`auto_detect` Whether this component automatically infers the options and schema for `CSV` and `NEWLINE_DELIMITED_JSON` sources. If this value is set to `false` and the destination table doesn’t exist, the output throws an insertion error as it is unable to insert data. > ⚠️ **CAUTION** > > This field delegates schema detection to the GCP BigQuery service. For the `CSV` format, values like `no` may be treated as booleans. **Type**: `bool` **Default**: `false` ### [](#batching)`batching` Configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#create_disposition)`create_disposition` Specifies the circumstances under which a destination table is created. - Use `CREATE_IF_NEEDED` to create the destination table if it does not already exist. Tables are created atomically on successful completion of a job. - Use `CREATE_NEVER` if the destination table must already exist. **Type**: `string` **Default**: `CREATE_IF_NEEDED` **Options**: `CREATE_IF_NEEDED`, `CREATE_NEVER` ### [](#credentials_json)`credentials_json` Sets the [Google Service Account Credentials JSON](https://developers.google.com/workspace/guides/create-credentials#create_credentials_for_a_service_account) (optional). > ⚠️ **WARNING** > > When using [interpolation functions](../../../configuration/interpolation/#bloblang-queries) to populate this field, wrap the function in single quotes, not double quotes. For example, use `'${secrets.GCP_CREDENTIALS_JSON}'` instead of `"${secrets.GCP_CREDENTIALS_JSON}"`. Double quotes cause JSON parsing errors because the credentials already contain JSON content. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#csv-2)`csv` Specify how CSV data is interpreted. **Type**: `object` ### [](#csv-allow_jagged_rows)`csv.allow_jagged_rows` Set to `true` to treat optional missing trailing columns as nulls in CSV data. **Type**: `bool` **Default**: `false` ### [](#csv-allow_quoted_newlines)`csv.allow_quoted_newlines` Whether quoted data sections containing new lines are allowed when reading CSV data. **Type**: `bool` **Default**: `false` ### [](#csv-encoding)`csv.encoding` The character encoding of CSV data. **Type**: `string` **Default**: `UTF-8` **Options**: `UTF-8`, `ISO-8859-1` ### [](#csv-field_delimiter)`csv.field_delimiter` The separator for fields in a CSV file. The output uses this value when reading or exporting data. **Type**: `string` **Default**: `,` ### [](#csv-header)`csv.header[]` A list of values to use as the header for each batch of messages. If not specified, the first line of each message is used as the header. **Type**: `array` **Default**: `[]` ### [](#csv-skip_leading_rows)`csv.skip_leading_rows` The number of rows at the top of a CSV file that BigQuery will skip when reading data. The default value is `1`, which allows Redpanda Connect to add the specified header in the first line of each batch sent to BigQuery. **Type**: `int` **Default**: `1` ### [](#dataset)`dataset` The BigQuery Dataset ID. **Type**: `string` ### [](#format-2)`format` The format of each incoming message. **Type**: `string` **Default**: `NEWLINE_DELIMITED_JSON` **Options**: `NEWLINE_DELIMITED_JSON`, `CSV`, `PARQUET` ### [](#ignore_unknown_values)`ignore_unknown_values` Set this value to `true` to ignore values that do not match the schema: - For the `CSV` format, extra values at the end of a line are ignored. - For the `NEWLINE_DELIMITED_JSON` format, values that do not match any column name are ignored. By default, this value is set to `false`, and records containing unknown values are treated as bad records. Use the `max_bad_records` field to customize how bad records are handled. **Type**: `bool` **Default**: `false` ### [](#job_labels)`job_labels` A list of labels to add to the load job. **Type**: `string` **Default**: `{}` ### [](#job_project)`job_project` Specify the project ID in which jobs are executed. If not set, the `project` value is used. **Type**: `string` **Default**: `""` ### [](#max_bad_records)`max_bad_records` The maximum number of bad records to ignore when reading data and [`ignore_unknown_values`](#ignore_unknown_values) is set to `true`. **Type**: `int` **Default**: `0` ### [](#max_in_flight)`max_in_flight` The maximum number of message batches to have in flight at a given time. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#project)`project` Specify the project ID of the dataset to insert data into. If not set, the project ID is inferred from the project linked to the service account or read from the `GOOGLE_CLOUD_PROJECT` environment variable. **Type**: `string` **Default**: `""` ### [](#table)`table` The table to insert messages into. **Type**: `string` ### [](#write_disposition)`write_disposition` Specifies how existing data in a destination table is treated. **Type**: `string` **Default**: `WRITE_APPEND` **Options**: `WRITE_APPEND`, `WRITE_EMPTY`, `WRITE_TRUNCATE` --- # Page 151: gcp_cloud_storage **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/gcp_cloud_storage.md --- # gcp\_cloud\_storage --- title: gcp_cloud_storage latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/gcp_cloud_storage page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/gcp_cloud_storage.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/gcp_cloud_storage.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/gcp_cloud_storage/)[Cache](/redpanda-connect/components/caches/gcp_cloud_storage/)[Input](/redpanda-connect/components/inputs/gcp_cloud_storage/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/gcp_cloud_storage/ "View the Cloud version of this component"), Self-Managed Sends message parts as objects to a Google Cloud Storage bucket. Each object is uploaded with the path specified with the `path` field. Introduced in version 3.43.0. #### Common ```yml outputs: label: "" gcp_cloud_storage: bucket: "" # No default (required) path: ${!counter()}-${!timestamp_unix_nano()}.txt content_type: application/octet-stream collision_mode: overwrite timeout: 3s credentials_json: "" max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" gcp_cloud_storage: bucket: "" # No default (required) path: ${!counter()}-${!timestamp_unix_nano()}.txt content_type: application/octet-stream content_encoding: "" collision_mode: overwrite chunk_size: 16777216 timeout: 3s credentials_json: "" max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` In order to have a different path for each object you should use function interpolations described in [Bloblang queries](../../../configuration/interpolation/#bloblang-queries), which are calculated per message of a batch. ## [](#metadata)Metadata Metadata fields on messages will be sent as headers, in order to mutate these values (or remove them) check out the [metadata docs](../../../configuration/metadata/). ## [](#credentials)Credentials By default Redpanda Connect will use a shared credentials file when connecting to GCP services. You can find out more in [Google Cloud Platform](../../../guides/cloud/gcp/). ## [](#batching)Batching It’s common to want to upload messages to Google Cloud Storage as batched archives, the easiest way to do this is to batch your messages at the output level and join the batch of messages with an [`archive`](../../processors/archive/) and/or [`compress`](../../processors/compress/) processor. For example, if we wished to upload messages as a .tar.gz archive of documents we could achieve that with the following config: ```yaml output: gcp_cloud_storage: bucket: TODO path: ${!counter()}-${!timestamp_unix_nano()}.tar.gz batching: count: 100 period: 10s processors: - archive: format: tar - compress: algorithm: gzip ``` Alternatively, if we wished to upload JSON documents as a single large document containing an array of objects we can do that with: ```yaml output: gcp_cloud_storage: bucket: TODO path: ${!counter()}-${!timestamp_unix_nano()}.json batching: count: 100 processors: - archive: format: json_array ``` ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](../../../configuration/batching/). ## [](#fields)Fields ### [](#batching-2)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#bucket)`bucket` The bucket to upload messages to. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#chunk_size)`chunk_size` An optional chunk size which controls the maximum number of bytes of the object that the Writer will attempt to send to the server in a single request. If ChunkSize is set to zero, chunking will be disabled. **Type**: `int` **Default**: `16777216` ### [](#collision_mode)`collision_mode` Determines how file path collisions should be dealt with. Options are "overwrite", which replaces the existing file with the new one, "append", which appends the message bytes to the original file, "error-if-exists", which returns an error and rejects the message if the file exists, and "ignore", does not modify the original file and drops the message. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). Requires version 3.53.0 or later. **Type**: `string` **Default**: `overwrite` **Options**: `overwrite`, `append`, `error-if-exists`, `ignore` ### [](#content_encoding)`content_encoding` An optional content encoding to set for each object. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#content_type)`content_type` The content type to set for each object. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `application/octet-stream` ### [](#credentials_json)`credentials_json` Base64-encoded Google Service Account credentials in JSON format (optional). Use this field to authenticate with Google Cloud services. For more information about creating service account credentials, see [Google’s service account documentation](https://developers.google.com/workspace/guides/create-credentials#create_credentials_for_a_service_account). This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of message batches to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#path)`path` The path of each message to upload. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `${!counter()}-${!timestamp_unix_nano()}.txt` ```yaml # Examples: path: ${!counter()}-${!timestamp_unix_nano()}.txt # --- path: ${!meta("kafka_key")}.json # --- path: ${!json("doc.namespace")}/${!json("doc.id")}.json ``` ### [](#timeout)`timeout` The maximum period to wait on an upload before abandoning it and reattempting. **Type**: `string` **Default**: `3s` ```yaml # Examples: timeout: 1s # --- timeout: 500ms ``` --- # Page 152: gcp_pubsub **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/gcp_pubsub.md --- # gcp\_pubsub --- title: gcp_pubsub latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/gcp_pubsub page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/gcp_pubsub.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/gcp_pubsub.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/gcp_pubsub/)[Input](/redpanda-connect/components/inputs/gcp_pubsub/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/gcp_pubsub/ "View the Cloud version of this component"), Self-Managed Sends messages to a GCP Cloud Pub/Sub topic. [Metadata](../../../configuration/metadata/) from messages are sent as attributes. #### Common ```yml outputs: label: "" gcp_pubsub: project: "" # No default (required) credentials_json: "" topic: "" # No default (required) endpoint: "" max_in_flight: 64 count_threshold: 100 delay_threshold: 10ms byte_threshold: 1000000 metadata: exclude_prefixes: [] batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" gcp_pubsub: project: "" # No default (required) credentials_json: "" topic: "" # No default (required) endpoint: "" ordering_key: "" # No default (optional) max_in_flight: 64 count_threshold: 100 delay_threshold: 10ms byte_threshold: 1000000 publish_timeout: 1m0s validate_topic: true metadata: exclude_prefixes: [] flow_control: max_outstanding_bytes: -1 max_outstanding_messages: 1000 limit_exceeded_behavior: block batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` For information on how to set up credentials, see [this guide](https://cloud.google.com/docs/authentication/production). ## [](#troubleshooting)Troubleshooting If you’re consistently seeing `Failed to send message to gcp_pubsub: context deadline exceeded` error logs without any further information it is possible that you are encountering [https://github.com/benthosdev/benthos/issues/1042](https://github.com/benthosdev/benthos/issues/1042), which occurs when metadata values contain characters that are not valid utf-8. This can frequently occur when consuming from Kafka as the key metadata field may be populated with an arbitrary binary value, but this issue is not exclusive to Kafka. If you are blocked by this issue then a work around is to delete either the specific problematic keys: ```yaml pipeline: processors: - mapping: | meta kafka_key = deleted() ``` Or delete all keys with: ```yaml pipeline: processors: - mapping: meta = deleted() ``` ## [](#fields)Fields ### [](#batching)`batching` Configures a batching policy on this output. While the PubSub client maintains its own internal buffering mechanism, preparing larger batches of messages can further trade-off some latency for throughput. **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#byte_threshold)`byte_threshold` Publish a batch when its size in bytes reaches this value. **Type**: `int` **Default**: `1000000` ### [](#count_threshold)`count_threshold` Publish a pubsub buffer when it has this many messages **Type**: `int` **Default**: `100` ### [](#credentials_json)`credentials_json` Base64-encoded Google Service Account credentials in JSON format (optional). Use this field to authenticate with Google Cloud services. For more information about creating service account credentials, see [Google’s service account documentation](https://developers.google.com/workspace/guides/create-credentials#create_credentials_for_a_service_account). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#delay_threshold)`delay_threshold` Publish a non-empty pubsub buffer after this delay has passed. **Type**: `string` **Default**: `10ms` ### [](#endpoint)`endpoint` An optional endpoint to override the default of `pubsub.googleapis.com:443`. This can be used to connect to a region specific pubsub endpoint. For a list of valid values, see [this document](https://cloud.google.com/pubsub/docs/reference/service_apis_overview#list_of_regional_endpoints). **Type**: `string` **Default**: `""` ```yaml # Examples: endpoint: us-central1-pubsub.googleapis.com:443 # --- endpoint: us-west3-pubsub.googleapis.com:443 ``` ### [](#flow_control)`flow_control` For a given topic, configures the PubSub client’s internal buffer for messages to be published. **Type**: `object` ### [](#flow_control-limit_exceeded_behavior)`flow_control.limit_exceeded_behavior` Configures the behavior when trying to publish additional messages while the flow controller is full. The available options are block (default), ignore (disable), and signal\_error (publish results will return an error). **Type**: `string` **Default**: `block` **Options**: `ignore`, `block`, `signal_error` ### [](#flow_control-max_outstanding_bytes)`flow_control.max_outstanding_bytes` Maximum size of buffered messages to be published. If less than or equal to zero, this is disabled. **Type**: `int` **Default**: `-1` ### [](#flow_control-max_outstanding_messages)`flow_control.max_outstanding_messages` Maximum number of buffered messages to be published. If less than or equal to zero, this is disabled. **Type**: `int` **Default**: `1000` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increasing this may improve throughput. **Type**: `int` **Default**: `64` ### [](#metadata)`metadata` Specify criteria for which metadata values are sent as attributes, all are sent by default. **Type**: `object` ### [](#metadata-exclude_prefixes)`metadata.exclude_prefixes[]` Provide a list of explicit metadata key prefixes to be excluded when adding metadata to sent messages. **Type**: `array` **Default**: `[]` ### [](#ordering_key)`ordering_key` The ordering key to use for publishing messages. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#project)`project` The project ID of the topic to publish to. **Type**: `string` ### [](#publish_timeout)`publish_timeout` The maximum length of time to wait before abandoning a publish attempt for a message. **Type**: `string` **Default**: `1m0s` ```yaml # Examples: publish_timeout: 10s # --- publish_timeout: 5m # --- publish_timeout: 60m ``` ### [](#topic)`topic` The topic to publish to. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#validate_topic)`validate_topic` Whether to validate the existence of the topic before publishing. If set to false and the topic does not exist, messages will be lost. **Type**: `bool` **Default**: `true` --- # Page 153: hdfs **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/hdfs.md --- # hdfs --- title: hdfs latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/hdfs page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/hdfs.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/hdfs.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/hdfs/)[Input](/redpanda-connect/components/inputs/hdfs/) **Available in:** Self-Managed Sends message parts as files to a HDFS directory. #### Common ```yml outputs: label: "" hdfs: hosts: [] # No default (required) user: "" directory: "" # No default (required) path: ${!counter()}-${!timestamp_unix_nano()}.txt max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" hdfs: hosts: [] # No default (required) user: "" directory: "" # No default (required) path: ${!counter()}-${!timestamp_unix_nano()}.txt max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` Each file is written with the path specified with the 'path' field, in order to have a different path for each object you should use function interpolations described [here](../../../configuration/interpolation/#bloblang-queries). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ## [](#fields)Fields ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#directory)`directory` A directory to store message files within. If the directory does not exist it will be created. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#hosts)`hosts[]` A list of target host addresses to connect to. **Type**: `array` ```yaml # Examples: hosts: localhost:9000 ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#path)`path` The path to upload messages as. Use interpolation functions to generate unique file paths. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `${!counter()}-${!timestamp_unix_nano()}.txt` ### [](#user)`user` A user ID to connect as. **Type**: `string` **Default**: `""` --- # Page 154: http_client **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/http_client.md --- # http\_client --- title: http_client latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/http_client page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/http_client.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/http_client.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/http_client/)[Input](/redpanda-connect/components/inputs/http_client/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/http_client/ "View the Cloud version of this component"), Self-Managed Sends messages to a HTTP server. #### Common ```yml outputs: label: "" http_client: url: "" # No default (required) verb: POST headers: {} rate_limit: "" # No default (optional) timeout: 5s max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" http_client: url: "" # No default (required) verb: POST headers: {} metadata: include_prefixes: [] include_patterns: [] dump_request_log_level: "" oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" oauth2: enabled: false client_key: "" client_secret: "" token_url: "" scopes: [] endpoint_params: {} basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] extract_headers: include_prefixes: [] include_patterns: [] rate_limit: "" # No default (optional) timeout: 5s retry_period: 1s max_retry_backoff: 300s retries: 3 follow_redirects: true backoff_on: - 429 drop_on: [] successful_on: [] proxy_url: "" # No default (optional) disable_http2: false batch_as_multipart: false propagate_response: false max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) multipart: [] ``` ## [](#message-sends)Message sends The body of the request sent to the HTTP server is the raw contents of the message payload. If the message has multiple parts (is a batch), the request is sent according to [RFC1341](https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html). To disable this behavior, set the [`batch_as_multipart`](#batch_as_multipart) field to `false`. When message retries are exhausted, this output rejects a message. Typically, a pipeline then continues attempts to send the message until it succeeds, whilst applying back pressure. ## [](#dynamic-url-and-header-settings)Dynamic URL and header settings You can set the [`url`](#url) and [`headers`](#headers) values dynamically using [function interpolations](../../../configuration/interpolation/#bloblang-queries). ## [](#propagate-responses)Propagate responses To propagate HTTP responses back to the input source, set the [`propagate_response`](#propagate_response) field to `true` . This feature is only available for inputs that support [synchronous responses](../../../guides/sync_responses/). ## [](#performance)Performance For improved performance, this output sends: - Multiple messages in parallel. Adjust the `max_in_flight` field value to tune the maximum number of in-flight messages (or message batches). - Messages as batches. You can configure batches at both input and output level. For more information, see [Message Batching](../../../configuration/batching/). ## [](#fields)Fields ### [](#backoff_on)`backoff_on[]` A list of status codes that indicate a request failure and trigger retries with an increasing backoff period between attempts. **Type**: `int` **Default**: ```yaml - 429 ``` ### [](#basic_auth)`basic_auth` Allows you to specify basic authentication. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#batch_as_multipart)`batch_as_multipart` When set to `true`, sends all message in a batch as a single request using [RFC1341](https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html). When set to `false`, sends messages in a batch as individual requests. **Type**: `bool` **Default**: `false` ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#disable_http2)`disable_http2` Whether to disable HTTP/2. By default, HTTP/2 is enabled. Requires version 4.44.0 or later. **Type**: `bool` **Default**: `false` ### [](#drop_on)`drop_on[]` A list of status codes that indicate a request failure where the input should not attempt retries. This helps avoid unnecessary retries for requests that are unlikely to succeed. > 📝 **NOTE** > > In these cases, the _request_ is dropped, but the _message_ that triggered the request is retained. **Type**: `int` **Default**: `[]` ### [](#dump_request_log_level)`dump_request_log_level` EXPERIMENTAL: Set the logging level for the request and response payloads of each HTTP request. Requires version 4.12.0 or later. **Type**: `string` **Default**: `""` **Options**: `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, `FATAL`, \`\` ### [](#extract_headers)`extract_headers` Specify which response headers to add to the resulting synchronous response messages as metadata. Header keys are automatically converted to lowercase before matching, so make sure that your patterns target the lowercase versions of the expected header keys. This field is only applicable when `propagate_response` is set to `true`. **Type**: `object` ### [](#extract_headers-include_patterns)`extract_headers.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#extract_headers-include_prefixes)`extract_headers.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#follow_redirects)`follow_redirects` Whether or not to transparently follow redirects, i.e. responses with 300-399 status codes. If disabled, the response message will contain the body, status, and headers from the redirect response and the processor will not make a request to the URL set in the Location header of the response. **Type**: `bool` **Default**: `true` ### [](#headers)`headers` A map of headers to add to the request. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `{}` ```yaml # Examples: headers: Content-Type: application/octet-stream traceparent: ${! tracing_span().traceparent } ``` ### [](#jwt)`jwt` Beta Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWT tokens provide secure, stateless authentication between services. **Type**: `object` ### [](#jwt-claims)`jwt.claims` A value used to identify the claims that issued the JWT. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` The cryptographic algorithm used to sign the JWT token. Supported algorithms include RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#max_retry_backoff)`max_retry_backoff` The maximum period to wait between failed requests. **Type**: `string` **Default**: `300s` ### [](#metadata)`metadata` Specify matching rules that determine which metadata keys to add to the HTTP request as headers (optional). **Type**: `object` ### [](#metadata-include_patterns)`metadata.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#metadata-include_prefixes)`metadata.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#multipart)`multipart[]` EXPERIMENTAL: Create explicit multipart HTTP requests by specifying an array of parts to add to a request. Each part consists of content headers and a data field, which can be populated dynamically. If populated, this field overrides the [default request creation behavior](#message-sends). Requires version 3.63.0 or later. **Type**: `object` **Default**: `[]` ### [](#multipart-body)`multipart[].body` The body of the individual message part. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ```yaml # Examples: body: ${! this.data.part1 } ``` ### [](#multipart-content_disposition)`multipart[].content_disposition` The content disposition of the individual message part. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ```yaml # Examples: content_disposition: form-data; name="bin"; filename='${! @AttachmentName } ``` ### [](#multipart-content_type)`multipart[].content_type` The content type of the individual message part. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ```yaml # Examples: content_type: application/bin ``` ### [](#oauth)`oauth` Configure OAuth version 1.0 authentication for secure API access. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` The value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` The secret that establishes ownership of the `oauth.access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` A value used to identify the client to the service provider. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#oauth2)`oauth2` Allows you to specify open authentication using OAuth version 2 and the client credentials token flow. **Type**: `object` ### [](#oauth2-client_key)`oauth2.client_key` A value used to identify the client to the token provider. **Type**: `string` **Default**: `""` ### [](#oauth2-client_secret)`oauth2.client_secret` The secret used to establish ownership of the client key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth2-enabled)`oauth2.enabled` Whether to use OAuth version 2 in requests. **Type**: `bool` **Default**: `false` ### [](#oauth2-endpoint_params)`oauth2.endpoint_params` A list of endpoint parameters specified as arrays of strings (optional). Requires version 4.21.0 or later. **Type**: `object` **Default**: `{}` ```yaml # Examples: endpoint_params: bar: - woof foo: - meow - quack ``` ### [](#oauth2-scopes)`oauth2.scopes[]` A list of requested permissions (optional). Requires version 3.45.0 or later. **Type**: `array` **Default**: `[]` ### [](#oauth2-token_url)`oauth2.token_url` The URL of the token provider. **Type**: `string` **Default**: `""` ### [](#propagate_response)`propagate_response` Whether to [propagate server responses back](../../../guides/sync_responses/) to the input. **Type**: `bool` **Default**: `false` ### [](#proxy_url)`proxy_url` A HTTP proxy URL (optional). **Type**: `string` ### [](#rate_limit)`rate_limit` A [rate limit](../../rate_limits/about/) to throttle requests by (optional). **Type**: `string` ### [](#retries)`retries` The maximum number of retry attempts to make. **Type**: `int` **Default**: `3` ### [](#retry_period)`retry_period` The initial period to wait between failed requests before retrying. **Type**: `string` **Default**: `1s` ### [](#successful_on)`successful_on[]` A list of HTTP status codes that should be considered as successful, even if they are not 2XX codes. This is useful for handling cases where non-2XX codes indicate that the request was processed successfully, such as `303 See Other` or `409 Conflict`. By default, all 2XX codes are considered successful unless they are specified in `backoff_on` or `drop_on` fields. **Type**: `int` **Default**: `[]` ### [](#timeout)`timeout` A static timeout to apply to requests. **Type**: `string` **Default**: `5s` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL to connect to. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#verb)`verb` A verb to connect with. **Type**: `string` **Default**: `POST` ```yaml # Examples: verb: POST # --- verb: GET # --- verb: DELETE ``` --- # Page 155: http_server **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/http_server.md --- # http\_server --- title: http_server latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/http_server page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/http_server.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/http_server.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/http_server/)[Input](/redpanda-connect/components/inputs/http_server/) **Available in:** Self-Managed Sets up an HTTP server that will send messages over HTTP(S) GET requests. HTTP 2.0 is supported when using TLS, which is enabled when key and cert files are specified. #### Common ```yml outputs: label: "" http_server: address: "" path: /get stream_path: /get/stream ws_path: /get/ws allowed_verbs: - "GET" ``` #### Advanced ```yml outputs: label: "" http_server: address: "" path: /get stream_path: /get/stream ws_path: /get/ws allowed_verbs: - "GET" timeout: 5s cert_file: "" key_file: "" cors: enabled: false allowed_origins: [] ``` Sets up an HTTP server that will send messages over HTTP(S) GET requests. If the `address` config field is left blank the [service-wide HTTP server](../../http/about/) will be used. Three endpoints will be registered at the paths specified by the fields `path`, `stream_path` and `ws_path`. Which allow you to consume a single message batch, a continuous stream of line delimited messages, or a websocket of messages for each request respectively. When messages are batched the `path` endpoint encodes the batch according to [RFC1341](https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html). This behavior can be overridden by [archiving your batches](../../../configuration/batching/#post-batch-processing). Please note, messages are considered delivered as soon as the data is written to the client. There is no concept of at least once delivery on this output. > ⚠️ **CAUTION: Endpoint caveats** > > Endpoint caveats > > Components within a Redpanda Connect config will register their respective endpoints in a non-deterministic order. This means that establishing precedence of endpoints that are registered via multiple `http_server` inputs or outputs (either within brokers or from cohabiting streams) is not possible in a predictable way. > > This ambiguity makes it difficult to ensure that paths which are both a subset of a path registered by a separate component, and end in a slash (`/`) and will therefore match against all extensions of that path, do not prevent the more specific path from matching against requests. > > It is therefore recommended that you ensure paths of separate components do not collide unless they are explicitly non-competing. > > For example, if you were to deploy two separate `http_server` inputs, one with a path `/foo/` and the other with a path `/foo/bar`, it would not be possible to ensure that the path `/foo/` does not swallow requests made to `/foo/bar`. ## [](#fields)Fields ### [](#address)`address` An alternative address to host from. If left empty the service wide address is used. **Type**: `string` **Default**: `""` ### [](#allowed_verbs)`allowed_verbs[]` An array of verbs that are allowed for the `path` and `stream_path` HTTP endpoint. **Type**: `array` **Default**: ```yaml - "GET" ``` ### [](#cert_file)`cert_file` Enable TLS by specifying a certificate and key file. Only valid with a custom `address`. **Type**: `string` **Default**: `""` ### [](#cors)`cors` Adds Cross-Origin Resource Sharing headers. Only valid with a custom `address`. Requires version 3.63.0 or later. **Type**: `object` ### [](#cors-allowed_origins)`cors.allowed_origins[]` An explicit list of origins that are allowed for CORS requests. **Type**: `array` **Default**: `[]` ### [](#cors-enabled)`cors.enabled` Whether to allow CORS requests. **Type**: `bool` **Default**: `false` ### [](#key_file)`key_file` Enable TLS by specifying a certificate and key file. Only valid with a custom `address`. **Type**: `string` **Default**: `""` ### [](#path)`path` The path from which discrete messages can be consumed. **Type**: `string` **Default**: `/get` ### [](#stream_path)`stream_path` The path from which a continuous stream of messages can be consumed. **Type**: `string` **Default**: `/get/stream` ### [](#timeout)`timeout` The maximum time to wait before a blocking, inactive connection is dropped (only applies to the `path` endpoint). **Type**: `string` **Default**: `5s` ### [](#ws_path)`ws_path` The path from which websocket connections can be established. **Type**: `string` **Default**: `/get/ws` --- # Page 156: iceberg **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/iceberg.md --- # iceberg --- title: iceberg latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/iceberg page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/iceberg.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/iceberg.adoc description: Fan out Redpanda topics to Apache Iceberg tables using the REST catalog API. page-git-created-date: "2026-03-05" page-git-modified-date: "2026-03-13" --- Fan out Redpanda topics to Apache Iceberg tables using the REST catalog API. Introduced in version 4.80.0. This output is well suited for migrating fanout pipelines from Kafka Connect to Redpanda Connect, and supports: - Multiple storage backends (S3, GCS, Azure) - Automatic table creation with schema detection - Partition transforms (year, month, day, hour, bucket, truncate) - Schema evolution (automatic column addition) - Transaction retry logic for concurrent writes ## [](#requirements)Requirements Ensure that your environment meets the following requirements: - This component is only available in cgo-enabled builds of Redpanda Connect (not in the Redpanda CLI or Docker image). You can either [download a prebuilt cgo-enabled binary](../../../install/prebuilt-binary/) or [build Redpanda Connect from source with cgo enabled](../../../install/build-from-source/). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](../../../configuration/batching/). ### Common ```yml outputs: label: "" iceberg: catalog: url: "" # No default (required) warehouse: "" # No default (optional) auth: oauth2: server_uri: /v1/oauth/tokens client_id: "" # No default (required) client_secret: "" # No default (required) scope: "" # No default (optional) bearer: "" # No default (optional) aws_sigv4: region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) service: "" # No default (optional) headers: "" # No default (optional) tls_skip_verify: false namespace: "" # No default (required) table: "" # No default (required) storage: aws_s3: bucket: "" # No default (required) region: "" # No default (optional) endpoint: "" # No default (optional) force_path_style_urls: false credentials: id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) gcp_cloud_storage: bucket: "" # No default (required) endpoint: "" # No default (optional) credentials_type: "" # No default (optional) credentials_file: "" # No default (optional) credentials_json: "" # No default (optional) azure_blob_storage: storage_account: "" # No default (required) container: "" # No default (required) endpoint: "" # No default (optional) storage_sas_token: "" # No default (optional) storage_connection_string: "" # No default (optional) storage_access_key: "" # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_in_flight: 4 ``` ### Advanced ```yml outputs: label: "" iceberg: catalog: url: "" # No default (required) warehouse: "" # No default (optional) auth: oauth2: server_uri: /v1/oauth/tokens client_id: "" # No default (required) client_secret: "" # No default (required) scope: "" # No default (optional) bearer: "" # No default (optional) aws_sigv4: region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) service: "" # No default (optional) headers: "" # No default (optional) tls_skip_verify: false namespace: "" # No default (required) table: "" # No default (required) storage: aws_s3: bucket: "" # No default (required) region: "" # No default (optional) endpoint: "" # No default (optional) force_path_style_urls: false credentials: id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) gcp_cloud_storage: bucket: "" # No default (required) endpoint: "" # No default (optional) credentials_type: "" # No default (optional) credentials_file: "" # No default (optional) credentials_json: "" # No default (optional) azure_blob_storage: storage_account: "" # No default (required) container: "" # No default (required) endpoint: "" # No default (optional) storage_sas_token: "" # No default (optional) storage_connection_string: "" # No default (optional) storage_access_key: "" # No default (optional) schema_evolution: enabled: false partition_spec: () table_location: "" # No default (optional) schema_metadata: "" new_column_type_mapping: "" # No default (optional) commit: manifest_merge_enabled: true max_snapshot_age: 24h max_retries: 3 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_in_flight: 4 ``` ## [](#catalog-integration)Catalog integration This output works with REST catalog implementations including Apache Polaris, AWS Glue Data Catalog, and Databricks Unity Catalog. ### [](#apache-polaris)Apache Polaris To use with [Apache Polaris](https://polaris.apache.org): - Set `catalog.url` to the Polaris REST endpoint (e.g., `[http://localhost:8181/api/catalog](http://localhost:8181/api/catalog)`). - Set `catalog.warehouse` to the catalog name configured in Polaris. - Configure `catalog.auth.oauth2` with client credentials granted access to the catalog. ### [](#aws-glue-data-catalog)AWS Glue Data Catalog To use with AWS Glue Data Catalog: - Set `catalog.url` to `[https://glue..amazonaws.com/iceberg](https://glue..amazonaws.com/iceberg)` (the REST client appends the API version automatically). - Set `catalog.warehouse` to your AWS account ID (the Glue catalog identifier). - Set `schema_evolution.table_location` to an S3 prefix (e.g., `s3://my-bucket/`) since Glue does not automatically assign table locations. - Configure `catalog.auth.aws_sigv4` with the appropriate region and set `service` to `glue`. - Configure `storage.aws_s3` with the same bucket and region. ### [](#azure-blob-storage-adls-gen2)Azure Blob Storage (ADLS Gen2) To use with Azure Data Lake Storage Gen2: - Configure `storage.azure_blob_storage` with your storage account name and container. - Authenticate using one of: `storage_access_key` (shared key), `storage_sas_token`, or `storage_connection_string`. - The storage account must have hierarchical namespace (HNS) enabled for ADLS Gen2 compatibility. ## [](#type-mapping)Type mapping | Bloblang type | Iceberg type | | --- | --- | | string | string | | bytes | binary | | bool | boolean | | number | double | | timestamp | timestamp (with timezone) | | object | struct | | array | list | ## [](#fields)Fields ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#catalog)`catalog` REST catalog configuration. **Type**: `object` ### [](#catalog-auth)`catalog.auth` Authentication configuration for the REST catalog. Only one authentication method can be active at a time. **Type**: `object` ### [](#catalog-auth-aws_sigv4)`catalog.auth.aws_sigv4` AWS SigV4 authentication (for AWS Glue Data Catalog or API Gateway). **Type**: `object` ### [](#catalog-auth-aws_sigv4-credentials)`catalog.auth.aws_sigv4.credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](../../../guides/cloud/aws/). **Type**: `object` ### [](#catalog-auth-aws_sigv4-credentials-from_ec2_role)`catalog.auth.aws_sigv4.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#catalog-auth-aws_sigv4-credentials-id)`catalog.auth.aws_sigv4.credentials.id` The ID of credentials to use. **Type**: `string` ### [](#catalog-auth-aws_sigv4-credentials-profile)`catalog.auth.aws_sigv4.credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#catalog-auth-aws_sigv4-credentials-role)`catalog.auth.aws_sigv4.credentials.role` A role ARN to assume. **Type**: `string` ### [](#catalog-auth-aws_sigv4-credentials-role_external_id)`catalog.auth.aws_sigv4.credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#catalog-auth-aws_sigv4-credentials-secret)`catalog.auth.aws_sigv4.credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#catalog-auth-aws_sigv4-credentials-token)`catalog.auth.aws_sigv4.credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#catalog-auth-aws_sigv4-endpoint)`catalog.auth.aws_sigv4.endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#catalog-auth-aws_sigv4-region)`catalog.auth.aws_sigv4.region` The AWS region to target. **Type**: `string` ### [](#catalog-auth-aws_sigv4-service)`catalog.auth.aws_sigv4.service` AWS service name for SigV4 signing. **Type**: `string` ### [](#catalog-auth-aws_sigv4-tcp)`catalog.auth.aws_sigv4.tcp` TCP socket configuration. **Type**: `object` ### [](#catalog-auth-aws_sigv4-tcp-connect_timeout)`catalog.auth.aws_sigv4.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#catalog-auth-aws_sigv4-tcp-keep_alive)`catalog.auth.aws_sigv4.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#catalog-auth-aws_sigv4-tcp-keep_alive-count)`catalog.auth.aws_sigv4.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#catalog-auth-aws_sigv4-tcp-keep_alive-idle)`catalog.auth.aws_sigv4.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#catalog-auth-aws_sigv4-tcp-keep_alive-interval)`catalog.auth.aws_sigv4.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#catalog-auth-aws_sigv4-tcp-tcp_user_timeout)`catalog.auth.aws_sigv4.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#catalog-auth-bearer)`catalog.auth.bearer` Static bearer token for authentication. For testing only, not recommended for production. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#catalog-auth-oauth2)`catalog.auth.oauth2` OAuth2 authentication configuration. **Type**: `object` ### [](#catalog-auth-oauth2-client_id)`catalog.auth.oauth2.client_id` OAuth2 client identifier. **Type**: `string` ### [](#catalog-auth-oauth2-client_secret)`catalog.auth.oauth2.client_secret` OAuth2 client secret. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#catalog-auth-oauth2-scope)`catalog.auth.oauth2.scope` OAuth2 scope to request. **Type**: `string` ### [](#catalog-auth-oauth2-server_uri)`catalog.auth.oauth2.server_uri` OAuth2 token endpoint URI. **Type**: `string` **Default**: `/v1/oauth/tokens` ### [](#catalog-headers)`catalog.headers` Custom HTTP headers to include in all requests to the catalog. **Type**: `object` ```yaml # Examples: headers: X-Api-Key: your-api-key ``` ### [](#catalog-tls_skip_verify)`catalog.tls_skip_verify` Skip TLS certificate verification. Not recommended for production. **Type**: `bool` **Default**: `false` ### [](#catalog-url)`catalog.url` The REST catalog endpoint URL. **Type**: `string` ```yaml # Examples: url: http://localhost:8181/api/catalog # --- url: https://polaris.example.com/api/catalog # --- url: https://glue.us-east-1.amazonaws.com/iceberg ``` ### [](#catalog-warehouse)`catalog.warehouse` The REST catalog warehouse. **Type**: `string` ```yaml # Examples: warehouse: redpanda-catalog ``` ### [](#commit)`commit` Commit behavior configuration. **Type**: `object` ### [](#commit-manifest_merge_enabled)`commit.manifest_merge_enabled` Merge small manifest files during commits to reduce metadata overhead. **Type**: `bool` **Default**: `true` ### [](#commit-max_retries)`commit.max_retries` Maximum number of times to retry a failed transaction commit. **Type**: `int` **Default**: `3` ### [](#commit-max_snapshot_age)`commit.max_snapshot_age` Maximum age of snapshots to retain for time-travel queries. Set to zero to disable removing old snapshots. **Type**: `string` **Default**: `24h` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `4` ### [](#namespace)`namespace` The Iceberg namespace for the table, dot delimiters are split as nested namespaces. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: namespace: analytics.events # --- namespace: production ``` ### [](#schema_evolution)`schema_evolution` Schema evolution configuration. **Type**: `object` ### [](#schema_evolution-enabled)`schema_evolution.enabled` Enable automatic schema evolution. When enabled, new columns will be automatically added to the table. **Type**: `bool` **Default**: `false` ### [](#schema_evolution-new_column_type_mapping)`schema_evolution.new_column_type_mapping` An optional Bloblang mapping to customize column types during schema evolution. This mapping is executed for each new column and can override the inferred or schema-metadata-derived type. The mapping receives an object with fields `name` (column name), `path` (dot-separated path), `value` (sample value), `inferred_type` (the type that would be used without this mapping), `message` (the full message body), `namespace`, and `table`. It must return a string with a valid Iceberg type name: `boolean`, `int`, `long`, `float`, `double`, `string`, `binary`, `date`, `time`, `timestamp`, `timestamptz`, `uuid`, `decimal(p,s)`, or `fixed[n]`. **Type**: `string` ### [](#schema_evolution-partition_spec)`schema_evolution.partition_spec` A Bloblang expression to evaluate when a new table is created to determine the table’s partition spec. The result of the mapping should be an iceberg partition spec in the same string format as the [^Redpanda Streaming Topic Property](https://docs.redpanda.com/current/manage/iceberg/about-iceberg-topics/#use-custom-partitioning) This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `()` ```yaml # Examples: partition_spec: (col1) # --- partition_spec: (nested.col) # --- partition_spec: (year(my_ts_col)) # --- partition_spec: (year(my_ts_col), col2) # --- partition_spec: (hour(my_ts_col), truncate(42, col2)) # --- partition_spec: (day(my_ts_col), bucket(4, nested.col)) # --- partition_spec: (day(my_ts_col), void(`non.nested column.with.dots`), identity(nested.column)) ``` ### [](#schema_evolution-schema_metadata)`schema_evolution.schema_metadata` The name of a message metadata field containing a schema definition. When set, the schema is used to determine column types during schema evolution and table creation instead of inferring types from values. The schema must be in the standard common schema format (the same format used by the `parquet_encode` processor’s `schema_metadata` field). For batches of messages, the first message’s schema is used. **Type**: `string` **Default**: `""` ### [](#schema_evolution-table_location)`schema_evolution.table_location` A prefix used as the location for new tables when the catalog does not automatically assign one. For example, AWS Glue requires explicit table locations. When set, table locations are derived as `{prefix}{namespace}/{table}`. **Type**: `string` ```yaml # Examples: table_location: s3://my-iceberg-bucket/ ``` ### [](#storage)`storage` Storage backend configuration for data files. Exactly one of `aws_s3`, `gcp_cloud_storage`, or `azure_blob_storage` must be specified. **Type**: `object` ### [](#storage-aws_s3)`storage.aws_s3` S3 storage configuration. **Type**: `object` ### [](#storage-aws_s3-bucket)`storage.aws_s3.bucket` The S3 bucket name. **Type**: `string` ```yaml # Examples: bucket: my-iceberg-data ``` ### [](#storage-aws_s3-credentials)`storage.aws_s3.credentials` Static AWS credentials for S3 access. When not specified, credentials are loaded from the default AWS credential chain. **Type**: `object` ### [](#storage-aws_s3-credentials-id)`storage.aws_s3.credentials.id` The AWS access key ID. **Type**: `string` ### [](#storage-aws_s3-credentials-secret)`storage.aws_s3.credentials.secret` The AWS secret access key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#storage-aws_s3-credentials-token)`storage.aws_s3.credentials.token` The AWS session token, required when using short term credentials. **Type**: `string` ### [](#storage-aws_s3-endpoint)`storage.aws_s3.endpoint` Custom endpoint for S3-compatible storage (e.g., MinIO). **Type**: `string` ```yaml # Examples: endpoint: http://localhost:9000 ``` ### [](#storage-aws_s3-force_path_style_urls)`storage.aws_s3.force_path_style_urls` Forces the client API to use path style URLs, which is often required when connecting to custom endpoints. **Type**: `bool` **Default**: `false` ### [](#storage-aws_s3-region)`storage.aws_s3.region` The AWS region. **Type**: `string` ```yaml # Examples: region: us-west-2 ``` ### [](#storage-azure_blob_storage)`storage.azure_blob_storage` Azure Blob Storage (ADLS Gen2) configuration. **Type**: `object` ### [](#storage-azure_blob_storage-container)`storage.azure_blob_storage.container` The Azure blob container name. **Type**: `string` ```yaml # Examples: container: iceberg-data ``` ### [](#storage-azure_blob_storage-endpoint)`storage.azure_blob_storage.endpoint` Custom endpoint for Azure-compatible storage. **Type**: `string` ### [](#storage-azure_blob_storage-storage_access_key)`storage.azure_blob_storage.storage_access_key` Azure storage access key for shared key authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#storage-azure_blob_storage-storage_account)`storage.azure_blob_storage.storage_account` The Azure storage account name. **Type**: `string` ```yaml # Examples: storage_account: mystorageaccount ``` ### [](#storage-azure_blob_storage-storage_connection_string)`storage.azure_blob_storage.storage_connection_string` Azure storage connection string. Use this or other auth methods, not both. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#storage-azure_blob_storage-storage_sas_token)`storage.azure_blob_storage.storage_sas_token` SAS token for authentication. Prefix with the container name followed by a dot if container-specific. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#storage-gcp_cloud_storage)`storage.gcp_cloud_storage` Google Cloud Storage configuration. **Type**: `object` ### [](#storage-gcp_cloud_storage-bucket)`storage.gcp_cloud_storage.bucket` The GCS bucket name. **Type**: `string` ```yaml # Examples: bucket: my-iceberg-data ``` ### [](#storage-gcp_cloud_storage-credentials_file)`storage.gcp_cloud_storage.credentials_file` Path to a GCP credentials JSON file. **Type**: `string` ### [](#storage-gcp_cloud_storage-credentials_json)`storage.gcp_cloud_storage.credentials_json` GCP credentials JSON content. Use this or `credentials_file`, not both. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#storage-gcp_cloud_storage-credentials_type)`storage.gcp_cloud_storage.credentials_type` The type of credentials to use. Valid values: `service_account`, `authorized_user`, `impersonated_service_account`, `external_account`. **Type**: `string` ```yaml # Examples: credentials_type: service_account ``` ### [](#storage-gcp_cloud_storage-endpoint)`storage.gcp_cloud_storage.endpoint` Custom endpoint for GCS-compatible storage. **Type**: `string` ### [](#table)`table` The Iceberg table name. Supports interpolation functions for dynamic table names. **Type**: `string` ```yaml # Examples: table: user_events # --- table: events_${!meta("topic")} ``` --- # Page 157: inproc **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/inproc.md --- # inproc --- title: inproc latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/inproc page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/inproc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/inproc.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/inproc/)[Input](/redpanda-connect/components/inputs/inproc/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/inproc/ "View the Cloud version of this component"), Self-Managed ```yml outputs: label: "" inproc: "" ``` Sends data directly to Redpanda Connect inputs by connecting to a unique ID. This allows you to hook up isolated streams whilst running Redpanda Connect in [streams mode](../../../guides/streams_mode/about/), it is NOT recommended that you connect the inputs of a stream with an output of the same stream, as feedback loops can lead to deadlocks in your message flow. It is possible to connect multiple inputs to the same inproc ID, resulting in messages dispatching in a round-robin fashion to connected inputs. However, only one output can assume an inproc ID, and will replace existing outputs if a collision occurs. --- # Page 158: kafka_franz **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/kafka_franz.md --- # kafka\_franz --- title: kafka_franz latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/kafka_franz page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/kafka_franz.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/kafka_franz.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/kafka_franz/)[Input](/redpanda-connect/components/inputs/kafka_franz/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/kafka_franz/ "View the Cloud version of this component"), Self-Managed > ⚠️ **WARNING: Deprecated in 4.68.0** > > Deprecated in 4.68.0 > > This component is deprecated and will be removed in the next major version release. Please consider moving onto the unified [`redpanda` input](../../inputs/redpanda/) and [`redpanda` output](../redpanda/) components. The `kafka_franz` output writes a batch of messages to Kafka brokers and waits for acknowledgement before propagating any acknowledgments back to the input. This output often outperforms the traditional `kafka` output, as well as providing more useful logs and error messages. This output uses the [Franz Kafka client library](https://github.com/twmb/franz-go). Introduced in version 3.61.0. #### Common ```yml outputs: label: "" kafka_franz: seed_brokers: [] # No default (required) topic: "" # No default (required) key: "" # No default (optional) partition: "" # No default (optional) metadata: include_prefixes: [] include_patterns: [] max_in_flight: 10 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" kafka_franz: seed_brokers: [] # No default (required) client_id: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) metadata_max_age: 1m request_timeout_overhead: 10s conn_idle_timeout: 20s tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s topic: "" # No default (required) key: "" # No default (optional) partition: "" # No default (optional) metadata: include_prefixes: [] include_patterns: [] timestamp_ms: "" # No default (optional) max_in_flight: 10 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) partitioner: "" # No default (optional) idempotent_write: true compression: "" # No default (optional) allow_auto_topic_creation: true timeout: 10s max_message_bytes: 1MiB broker_write_max_bytes: 100MiB ``` ## [](#fields)Fields ### [](#allow_auto_topic_creation)`allow_auto_topic_creation` Enables topics to be auto created if they do not exist when fetching their metadata. **Type**: `bool` **Default**: `true` ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#broker_write_max_bytes)`broker_write_max_bytes` The maximum number of bytes this output can write to a broker connection in a single write. This field corresponds to Kafka’s `socket.request.max.bytes`. **Type**: `string` **Default**: `100MiB` ```yaml # Examples: broker_write_max_bytes: 128MB # --- broker_write_max_bytes: 50mib ``` ### [](#client_id)`client_id` An identifier for the client connection. **Type**: `string` **Default**: `redpanda-connect` ### [](#compression)`compression` Set an explicit compression type (optional). The default preference is to use `snappy` when the broker supports it. Otherwise, use `none`. **Type**: `string` **Options**: `lz4`, `snappy`, `gzip`, `none`, `zstd` ### [](#conn_idle_timeout)`conn_idle_timeout` The maximum duration that connections can remain idle before they are automatically closed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `20s` ### [](#idempotent_write)`idempotent_write` Enables the idempotent write producer option. This requires the `IDEMPOTENT_WRITE` permission on `CLUSTER`. Disable this option if the `IDEMPOTENT_WRITE` permission is unavailable. **Type**: `bool` **Default**: `true` ### [](#key)`key` An optional key to populate for each message. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of batches to send in parallel at any given time. **Type**: `int` **Default**: `10` ### [](#max_message_bytes)`max_message_bytes` The maximum space (in bytes) that an individual message may use. Messages larger than this value are rejected. This field corresponds to Kafka’s `max.message.bytes`. **Type**: `string` **Default**: `1MiB` ```yaml # Examples: max_message_bytes: 100MB # --- max_message_bytes: 50mib ``` ### [](#metadata)`metadata` Configure which metadata values are added to messages as headers. This allows you to pass additional context information along with your messages. **Type**: `object` ### [](#metadata-include_patterns)`metadata.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#metadata-include_prefixes)`metadata.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#metadata_max_age)`metadata_max_age` The maximum period of time after which metadata is refreshed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Lower values provide more responsive topic and partition discovery but may increase broker load. Higher values reduce broker queries but can delay detection of topology changes. **Type**: `string` **Default**: `1m` ### [](#partition)`partition` Set a partition for each message (optional). This field is only relevant when the `partitioner` is set to `manual`. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). You must provide an interpolation string that is a valid integer. **Type**: `string` ```yaml # Examples: partition: ${! meta("partition") } ``` ### [](#partitioner)`partitioner` Override the default murmur2 hashing partitioner. **Type**: `string` | Option | Summary | | --- | --- | | least_backup | Chooses the least backed up partition (the partition with the fewest amount of buffered records). Partitions are selected per batch. | | manual | Manually select a partition for each message, requires the field partition to be specified. | | murmur2_hash | Kafka’s default hash algorithm that uses a 32-bit murmur2 hash of the key to compute which partition the record will be on. | | round_robin | Round-robin’s messages through all available partitions. This algorithm has lower throughput and causes higher CPU load on brokers, but can be useful if you want to ensure an even distribution of records to partitions. | ### [](#request_timeout_overhead)`request_timeout_overhead` Grants an additional buffer or overhead to requests that have timeout fields defined. This field is based on the behavior of Apache Kafka’s `request.timeout.ms` parameter, but with the option to extend the timeout deadline. **Type**: `string` **Default**: `10s` ### [](#sasl)`sasl[]` Specify one or more methods or mechanisms of SASL authentication, which are attempted in order. If the broker supports the first SASL mechanism, all connections use it. If the first mechanism fails, the client picks the first supported mechanism. If the broker does not support any client mechanisms, all connections fail. **Type**: `object` ```yaml # Examples: sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ### [](#sasl-aws)`sasl[].aws` Contains AWS specific fields for when the `mechanism` is set to `AWS_MSK_IAM`. **Type**: `object` ### [](#sasl-aws-credentials)`sasl[].aws.credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](../../../guides/cloud/aws/). **Type**: `object` ### [](#sasl-aws-credentials-from_ec2_role)`sasl[].aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#sasl-aws-credentials-id)`sasl[].aws.credentials.id` The ID of credentials to use. **Type**: `string` ### [](#sasl-aws-credentials-profile)`sasl[].aws.credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#sasl-aws-credentials-role)`sasl[].aws.credentials.role` A role ARN to assume. **Type**: `string` ### [](#sasl-aws-credentials-role_external_id)`sasl[].aws.credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#sasl-aws-credentials-secret)`sasl[].aws.credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#sasl-aws-credentials-token)`sasl[].aws.credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#sasl-aws-endpoint)`sasl[].aws.endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#sasl-aws-region)`sasl[].aws.region` The AWS region to target. **Type**: `string` ### [](#sasl-aws-tcp)`sasl[].aws.tcp` TCP socket configuration. **Type**: `object` ### [](#sasl-aws-tcp-connect_timeout)`sasl[].aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-aws-tcp-keep_alive)`sasl[].aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#sasl-aws-tcp-keep_alive-count)`sasl[].aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#sasl-aws-tcp-keep_alive-idle)`sasl[].aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-keep_alive-interval)`sasl[].aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-tcp_user_timeout)`sasl[].aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-extensions)`sasl[].extensions` Key/value pairs to add to OAUTHBEARER authentication requests. **Type**: `string` ### [](#sasl-mechanism)`sasl[].mechanism` The SASL mechanism to use. **Type**: `string` | Option | Summary | | --- | --- | | AWS_MSK_IAM | AWS IAM based authentication as specified by the 'aws-msk-iam-auth' java library. | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. | | REDPANDA_CLOUD_SERVICE_ACCOUNT | Redpanda Cloud Service Account authentication when running in Redpanda Cloud. | | SCRAM-SHA-256 | SCRAM based authentication as specified in RFC5802. | | SCRAM-SHA-512 | SCRAM based authentication as specified in RFC5802. | | none | Disable sasl authentication | ### [](#sasl-password)`sasl[].password` A password to provide for PLAIN or SCRAM-\* authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#sasl-token)`sasl[].token` The token to use for a single session’s OAUTHBEARER authentication. **Type**: `string` **Default**: `""` ### [](#sasl-username)`sasl[].username` A username to provide for PLAIN or SCRAM-\* authentication. **Type**: `string` **Default**: `""` ### [](#seed_brokers)`seed_brokers[]` A list of broker addresses to connect to in order. Use commas to separate multiple addresses in a single list item. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" # --- seed_brokers: - "foo:9092" - "bar:9092" # --- seed_brokers: - "foo:9092,bar:9092" ``` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` The maximum period of time to wait for message sends before abandoning the request and retrying. **Type**: `string` **Default**: `10s` ### [](#timestamp_ms)`timestamp_ms` Set a timestamp (in milliseconds) for each message (optional). When left empty, the current timestamp is used. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: timestamp_ms: ${! timestamp_unix_milli() } # --- timestamp_ms: ${! metadata("kafka_timestamp_ms") } ``` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic)`topic` A topic to write messages to. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 159: kafka **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/kafka.md --- # kafka --- title: kafka latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/kafka page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/kafka.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/kafka.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/kafka/)[Input](/redpanda-connect/components/inputs/kafka/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/kafka/ "View the Cloud version of this component"), Self-Managed > ⚠️ **WARNING: Deprecated in 4.68.0** > > Deprecated in 4.68.0 > > This component is deprecated and will be removed in the next major version release. Please consider moving onto the unified [`redpanda` input](../../inputs/redpanda/) and [`redpanda` output](../redpanda/) components. The `kafka` output writes a batch of messages to Kafka brokers and waits for acknowledgement before propagating any acknowledgements back to the input. #### Common ```yml outputs: label: "" kafka: addresses: [] # No default (required) topic: "" # No default (required) target_version: "" # No default (optional) key: "" partitioner: fnv1a_hash compression: none static_headers: "" # No default (optional) metadata: exclude_prefixes: [] max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" kafka: addresses: [] # No default (required) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: mechanism: none user: "" password: "" access_token: "" token_cache: "" token_key: "" topic: "" # No default (required) client_id: benthos target_version: "" # No default (optional) rack_id: "" key: "" partitioner: fnv1a_hash partition: "" custom_topic_creation: enabled: false partitions: -1 replication_factor: -1 compression: none static_headers: "" # No default (optional) metadata: exclude_prefixes: [] inject_tracing_map: "" # No default (optional) max_in_flight: 64 idempotent_write: false ack_replicas: false max_msg_bytes: 1000000 timeout: 5s retry_as_batch: false batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_retries: 0 backoff: initial_interval: 3s max_interval: 10s max_elapsed_time: 30s timestamp_ms: "" # No default (optional) ``` The configuration field `ack_replicas` determines whether Redpanda Connect waits for acknowledgement from all replicas or just a single broker. Both the `key` and `topic` fields can be dynamically set using function interpolations described in [Bloblang queries](../../../configuration/interpolation/#bloblang-queries). [Metadata](../../../configuration/metadata/) will be added to each message sent as headers (version 0.11+), but can be restricted using the field [`metadata`](#metadata). ## [](#strict-ordering-and-retries)Strict ordering and retries When strict ordering is required for messages written to topic partitions it is important to ensure that both the field `max_in_flight` is set to `1` and that the field `retry_as_batch` is set to `true`. You must also ensure that failed batches are never rerouted back to the same output. This can be done by setting the field `max_retries` to `0` and `backoff.max_elapsed_time` to empty, which will apply back pressure indefinitely until the batch is sent successfully. However, this also means that manual intervention will eventually be required in cases where the batch cannot be sent due to configuration problems such as an incorrect `max_msg_bytes` estimate. A less strict but automated alternative would be to route failed batches to a dead letter queue using a [`fallback` broker](../fallback/), but this would allow subsequent batches to be delivered in the meantime whilst those failed batches are dealt with. ## [](#troubleshooting)Troubleshooting If you’re seeing issues writing to or reading from Kafka with this component then it’s worth trying out the newer [`kafka_franz` output](../kafka_franz/). - I’m seeing logs that report `Failed to connect to kafka: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)`, but the brokers are definitely reachable. Unfortunately this error message will appear for a wide range of connection problems even when the broker endpoint can be reached. Double check your authentication configuration and also ensure that you have [enabled TLS](#tlsenabled) if applicable. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](../../../configuration/batching/). ## [](#fields)Fields ### [](#ack_replicas)`ack_replicas` Ensure that messages have been copied across all replicas before acknowledging receipt. **Type**: `bool` **Default**: `false` ### [](#addresses)`addresses[]` A list of broker addresses to connect to. If an item of the list contains commas it will be expanded into multiple addresses. **Type**: `array` ```yaml # Examples: addresses: - "localhost:9092" # --- addresses: - "localhost:9041,localhost:9042" # --- addresses: - "localhost:9041" - "localhost:9042" ``` ### [](#backoff)`backoff` Control time intervals between retry attempts. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` The initial period to wait between retry attempts. The retry interval increases for each failed attempt, up to the `backoff.max_interval` value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `3s` ```yaml # Examples: initial_interval: 50ms # --- initial_interval: 1s ``` ### [](#backoff-max_elapsed_time)`backoff.max_elapsed_time` The maximum overall period of time to spend on retry attempts before the request is aborted. Setting this value to a zeroed duration (such as `0s`) will result in unbounded retries. **Type**: `string` **Default**: `30s` ```yaml # Examples: max_elapsed_time: 1m # --- max_elapsed_time: 1h ``` ### [](#backoff-max_interval)`backoff.max_interval` The maximum period to wait between retry attempts **Type**: `string` **Default**: `10s` ```yaml # Examples: max_interval: 5s # --- max_interval: 1m ``` ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#client_id)`client_id` An identifier for the client connection. **Type**: `string` **Default**: `benthos` ### [](#compression)`compression` The compression algorithm to use. **Type**: `string` **Default**: `none` **Options**: `none`, `snappy`, `lz4`, `gzip`, `zstd` ### [](#custom_topic_creation)`custom_topic_creation` If enabled, topics will be created with the specified number of partitions and replication factor if they do not already exist. **Type**: `object` ### [](#custom_topic_creation-enabled)`custom_topic_creation.enabled` Whether to enable custom topic creation. **Type**: `bool` **Default**: `false` ### [](#custom_topic_creation-partitions)`custom_topic_creation.partitions` The number of partitions to create for new topics. Leave at -1 to use the broker configured default. Must be >= 1. **Type**: `int` **Default**: `-1` ### [](#custom_topic_creation-replication_factor)`custom_topic_creation.replication_factor` The replication factor to use for new topics. Leave at -1 to use the broker configured default. Must be an odd number, and less then or equal to the number of brokers. **Type**: `int` **Default**: `-1` ### [](#idempotent_write)`idempotent_write` Enable the idempotent write producer option. This requires the `IDEMPOTENT_WRITE` permission on `CLUSTER` and can be disabled if this permission is not available. **Type**: `bool` **Default**: `false` ### [](#inject_tracing_map)`inject_tracing_map` EXPERIMENTAL: A [Bloblang mapping](../../../guides/bloblang/about/) used to inject an object containing tracing propagation information into outbound messages. The specification of the injected fields will match the format used by the service wide tracer. Requires version 3.45.0 or later. **Type**: `string` ```yaml # Examples: inject_tracing_map: meta = @.merge(this) # --- inject_tracing_map: root.meta.span = this ``` ### [](#key)`key` An optional key to populate for each message. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#max_msg_bytes)`max_msg_bytes` The maximum size in bytes of messages sent to the target topic. **Type**: `int` **Default**: `1000000` ### [](#max_retries)`max_retries` The maximum number of retries before giving up on the request. If set to zero there is no discrete limit. **Type**: `int` **Default**: `0` ### [](#metadata)`metadata` Specify criteria for which metadata values are sent with messages as headers. **Type**: `object` ### [](#metadata-exclude_prefixes)`metadata.exclude_prefixes[]` Provide a list of explicit metadata key prefixes to be excluded when adding metadata to sent messages. **Type**: `array` **Default**: `[]` ### [](#partition)`partition` The manually-specified partition to publish messages to, relevant only when the field `partitioner` is set to `manual`. Must be able to parse as a 32-bit integer. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#partitioner)`partitioner` The partitioning algorithm to use. **Type**: `string` **Default**: `fnv1a_hash` **Options**: `fnv1a_hash`, `murmur2_hash`, `random`, `round_robin`, `manual` ### [](#rack_id)`rack_id` A rack identifier for this client. **Type**: `string` **Default**: `""` ### [](#retry_as_batch)`retry_as_batch` When enabled forces an entire batch of messages to be retried if any individual message fails on a send, otherwise only the individual messages that failed are retried. Disabling this helps to reduce message duplicates during intermittent errors, but also makes it impossible to guarantee strict ordering of messages. **Type**: `bool` **Default**: `false` ### [](#sasl)`sasl` Enables SASL authentication. **Type**: `object` ### [](#sasl-access_token)`sasl.access_token` A static OAUTHBEARER access token **Type**: `string` **Default**: `""` ### [](#sasl-mechanism)`sasl.mechanism` The SASL authentication mechanism, if left empty SASL authentication is not used. **Type**: `string` **Default**: `none` | Option | Summary | | --- | --- | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. NOTE: When using plain text auth it is extremely likely that you’ll also need to enable TLS. | | SCRAM-SHA-256 | Authentication using the SCRAM-SHA-256 mechanism. | | SCRAM-SHA-512 | Authentication using the SCRAM-SHA-512 mechanism. | | none | Default, no SASL authentication. | ### [](#sasl-password)`sasl.password` A PLAIN password. It is recommended that you use environment variables to populate this field. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: ${PASSWORD} ``` ### [](#sasl-token_cache)`sasl.token_cache` Instead of using a static `access_token` allows you to query a [`cache`](../../caches/about/) resource to fetch OAUTHBEARER tokens from **Type**: `string` **Default**: `""` ### [](#sasl-token_key)`sasl.token_key` Required when using a `token_cache`, the key to query the cache with for tokens. **Type**: `string` **Default**: `""` ### [](#sasl-user)`sasl.user` A PLAIN username. It is recommended that you use environment variables to populate this field. **Type**: `string` **Default**: `""` ```yaml # Examples: user: ${USER} ``` ### [](#static_headers)`static_headers` An optional map of static headers that should be added to messages in addition to metadata. **Type**: `string` ```yaml # Examples: static_headers: first-static-header: value-1 second-static-header: value-2 ``` ### [](#target_version)`target_version` The version of the Kafka protocol to use. This limits the capabilities used by the client and should ideally match the version of your brokers. Defaults to the oldest supported stable version. **Type**: `string` ```yaml # Examples: target_version: 2.1.0 # --- target_version: 3.1.0 ``` ### [](#timeout)`timeout` The maximum period of time to wait for message sends before abandoning the request and retrying. **Type**: `string` **Default**: `5s` ### [](#timestamp_ms)`timestamp_ms` Set a timestamp (in milliseconds) for each message (optional). When left empty, the current timestamp is used. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: timestamp_ms: ${! timestamp_unix_milli() } # --- timestamp_ms: ${! metadata("kafka_timestamp_ms") } ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#topic)`topic` The topic to publish messages to. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 160: mongodb **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/mongodb.md --- # mongodb --- title: mongodb latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/mongodb page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/mongodb.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/mongodb.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/mongodb/)[Cache](/redpanda-connect/components/caches/mongodb/)[Input](/redpanda-connect/components/inputs/mongodb/)[Processor](/redpanda-connect/components/processors/mongodb/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/mongodb/ "View the Cloud version of this component"), Self-Managed Inserts items into a MongoDB collection. Introduced in version 3.43.0. #### Common ```yml outputs: label: "" mongodb: url: "" # No default (required) database: "" # No default (required) username: "" password: "" collection: "" # No default (required) operation: update-one write_concern: w: majority j: false w_timeout: "" document_map: "" filter_map: "" hint_map: "" upsert: false max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" mongodb: url: "" # No default (required) database: "" # No default (required) username: "" password: "" app_name: benthos collection: "" # No default (required) operation: update-one write_concern: w: majority j: false w_timeout: "" document_map: "" filter_map: "" hint_map: "" upsert: false max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#performance)Performance This output benefits from sending multiple messages in flight, in parallel, for improved performance. You can tune the maximum number of in flight messages (or message batches) using the `max_in_flight` field. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. For more information, see [Message Batching](../../../configuration/batching/). ## [](#fields)Fields ### [](#app_name)`app_name` The client application name. **Type**: `string` **Default**: `benthos` ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#collection)`collection` The name of the target collection. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#database)`database` The name of the target MongoDB database. **Type**: `string` ### [](#document_map)`document_map` A Bloblang map that represents a document to store in MongoDB, expressed as [extended JSON in canonical form](https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/). The `document_map` parameter is required for the following database operations: `insert-one`, `replace-one`, and `update-one`. **Type**: `string` **Default**: `""` ```yaml # Examples: document_map: |- root.a = this.foo root.b = this.bar ``` ### [](#filter_map)`filter_map` A Bloblang map that represents a filter for a MongoDB command, expressed as [extended JSON in canonical form](https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/). The `filter_map` parameter is required for all database operations except `insert-one`. This output uses `filter_map` to find documents for the specified operation. For example, for a `delete-one` operation, the filter map should include the fields required to locate the document for deletion. **Type**: `string` **Default**: `""` ```yaml # Examples: filter_map: |- root.a = this.foo root.b = this.bar ``` ### [](#hint_map)`hint_map` A Bloblang map that represents a hint or index for a MongoDB command to use, expressed as [extended JSON in canonical form](https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/). This map is optional, and is used with all operations except `insert-one`. Define a `hint_map` to improve performance when finding documents in the MongoDB database. **Type**: `string` **Default**: `""` ```yaml # Examples: hint_map: |- root.a = this.foo root.b = this.bar ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this number to improve throughput. **Type**: `int` **Default**: `64` ### [](#operation)`operation` The MongoDB database operation to perform. **Type**: `string` **Default**: `update-one` **Options**: `insert-one`, `delete-one`, `delete-many`, `replace-one`, `update-one` ### [](#password)`password` The password to use for authentication. Used together with `username` for basic authentication or with encrypted private keys for secure access. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#upsert)`upsert` The `upsert` parameter is optional, and only applies for `update-one` and `replace-one` operations. If the filter specified in `filter_map` matches an existing document, this operation updates or replaces the document, otherwise a new document is created. Requires version 3.60.0 or later. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target MongoDB server. **Type**: `string` ```yaml # Examples: url: mongodb://localhost:27017 ``` ### [](#username)`username` The username required to connect to the database. **Type**: `string` **Default**: `""` ### [](#write_concern)`write_concern` The [write concern settings](https://www.mongodb.com/docs/manual/reference/write-concern/) for the MongoDB connection. **Type**: `object` ### [](#write_concern-j)`write_concern.j` The `j` requests acknowledgement from MongoDB, which is created when write operations are written to the journal. **Type**: `bool` **Default**: `false` ### [](#write_concern-w)`write_concern.w` The `w` requests acknowledgement, which write operations propagate to the specified number of MongoDB instances. **Type**: `string` **Default**: `majority` ### [](#write_concern-w_timeout)`write_concern.w_timeout` The write concern timeout. **Type**: `string` **Default**: `""` --- # Page 161: mqtt **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/mqtt.md --- # mqtt --- title: mqtt latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/mqtt page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/mqtt.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/mqtt.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/mqtt/)[Input](/redpanda-connect/components/inputs/mqtt/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/mqtt/ "View the Cloud version of this component"), Self-Managed Pushes messages to an MQTT broker. Introduced in version 4.37.0. #### Common ```yml outputs: label: "" mqtt: urls: [] # No default (required) client_id: "" connect_timeout: 30s topic: "" # No default (required) qos: 1 write_timeout: 3s retained: false max_in_flight: 64 ``` #### Advanced ```yml outputs: label: "" mqtt: urls: [] # No default (required) client_id: "" dynamic_client_id_suffix: "" # No default (optional) connect_timeout: 30s will: enabled: false qos: 0 retained: false topic: "" payload: "" user: "" password: "" keepalive: 30 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] topic: "" # No default (required) qos: 1 write_timeout: 3s retained: false retained_interpolated: "" # No default (optional) max_in_flight: 64 ``` The `topic` field can be dynamically set using function interpolations described [here](../../../configuration/interpolation/#bloblang-queries). When sending batched messages these interpolations are performed per message part. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ## [](#fields)Fields ### [](#client_id)`client_id` An identifier for the client connection. **Type**: `string` **Default**: `""` ### [](#connect_timeout)`connect_timeout` The maximum amount of time to wait in order to establish a connection before the attempt is abandoned. Requires version 3.58.0 or later. **Type**: `string` **Default**: `30s` ```yaml # Examples: connect_timeout: 1s # --- connect_timeout: 500ms ``` ### [](#dynamic_client_id_suffix)`dynamic_client_id_suffix` Append a dynamically generated suffix to the specified `client_id` on each run of the pipeline. This can be useful when clustering Redpanda Connect producers. **Type**: `string` | Option | Summary | | --- | --- | | nanoid | append a nanoid of length 21 characters | ### [](#keepalive)`keepalive` Max seconds of inactivity before a keepalive message is sent. **Type**: `int` **Default**: `30` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#password)`password` A password to connect with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#qos)`qos` The QoS value to set for each message. Has options 0, 1, 2. **Type**: `int` **Default**: `1` ### [](#retained)`retained` Set message as retained on the topic. **Type**: `bool` **Default**: `false` ### [](#retained_interpolated)`retained_interpolated` Override the value of `retained` with an interpolable value, this allows it to be dynamically set based on message contents. The value must resolve to either `true` or `false`. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). Requires version 3.59.0 or later. **Type**: `string` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#topic)`topic` The topic to publish messages to. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#urls)`urls[]` A list of URLs to connect to. Use the format `scheme://host:port`, where: - `scheme` is one of the following: `tcp`, `ssl`, `ws` - `host` is the IP address or hostname - `port` is the port on which the MQTT broker accepts connections If an item in the list contains commas, it is expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "tcp://localhost:1883" ``` ### [](#user)`user` A username to connect with. **Type**: `string` **Default**: `""` ### [](#will)`will` Set last will message in case of Redpanda Connect failure **Type**: `object` ### [](#will-enabled)`will.enabled` Whether to enable last will messages. **Type**: `bool` **Default**: `false` ### [](#will-payload)`will.payload` Set payload for last will message. **Type**: `string` **Default**: `""` ### [](#will-qos)`will.qos` Set QoS for last will message. Valid values are: 0, 1, 2. **Type**: `int` **Default**: `0` ### [](#will-retained)`will.retained` Set retained for last will message. **Type**: `bool` **Default**: `false` ### [](#will-topic)`will.topic` Set topic for last will message. **Type**: `string` **Default**: `""` ### [](#write_timeout)`write_timeout` The maximum amount of time to wait to write data before the attempt is abandoned. Requires version 3.58.0 or later. **Type**: `string` **Default**: `3s` ```yaml # Examples: write_timeout: 1s # --- write_timeout: 500ms ``` --- # Page 162: nanomsg **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/nanomsg.md --- # nanomsg --- title: nanomsg latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/nanomsg page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/nanomsg.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/nanomsg.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/nanomsg/)[Input](/redpanda-connect/components/inputs/nanomsg/) **Available in:** Self-Managed Send messages over a Nanomsg socket. #### Common ```yml outputs: label: "" nanomsg: urls: [] # No default (required) bind: false socket_type: PUSH poll_timeout: 5s max_in_flight: 64 ``` #### Advanced ```yml outputs: label: "" nanomsg: urls: [] # No default (required) bind: false socket_type: PUSH poll_timeout: 5s max_in_flight: 64 ``` Currently only PUSH and PUB sockets are supported. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ## [](#fields)Fields ### [](#bind)`bind` Whether the URLs listed should be bind (otherwise they are connected to). **Type**: `bool` **Default**: `false` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#poll_timeout)`poll_timeout` The maximum period of time to wait for a message to send before the request is abandoned and reattempted. **Type**: `string` **Default**: `5s` ### [](#socket_type)`socket_type` The socket type to send with. **Type**: `string` **Default**: `PUSH` **Options**: `PUSH`, `PUB` ### [](#urls)`urls[]` A list of URLs to connect to. If an item of the list contains commas it will be expanded into multiple URLs. **Type**: `array` --- # Page 163: nats_jetstream **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/nats_jetstream.md --- # nats\_jetstream --- title: nats_jetstream latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/nats_jetstream page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/nats_jetstream.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/nats_jetstream.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/nats_jetstream/)[Input](/redpanda-connect/components/inputs/nats_jetstream/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/nats_jetstream/ "View the Cloud version of this component"), Self-Managed Write messages to a NATS JetStream subject. Introduced in version 3.46.0. #### Common ```yml outputs: label: "" nats_jetstream: urls: [] # No default (required) subject: "" # No default (required) headers: {} metadata: include_prefixes: [] include_patterns: [] max_in_flight: 1024 ``` #### Advanced ```yml outputs: label: "" nats_jetstream: urls: [] # No default (required) max_reconnects: "" # No default (optional) subject: "" # No default (required) headers: {} metadata: include_prefixes: [] include_patterns: [] max_in_flight: 1024 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) inject_tracing_map: "" # No default (optional) ``` ## [](#connection-name)Connection name When monitoring and managing a production [NATS system](https://docs.nats.io/nats-concepts/overview), it is often useful to know which connection a message was sent or received from. To achieve this, set the connection name option when creating a NATS connection. Redpanda Connect can then automatically set the connection name to the NATS component label, so that monitoring tools between NATS and Redpanda Connect can stay in sync. ## [](#authentication)Authentication A number of Redpanda Connect components use NATS services. Each of these components support optional, advanced authentication parameters for [NKeys](https://docs.nats.io/nats-server/configuration/securing_nats/auth_intro/nkey_auth) and [user credentials](https://docs.nats.io/using-nats/developer/connecting/creds). For an in-depth guide, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/nats_admin/security/jwt). ### [](#nkeys)NKeys NATS server can use NKeys in several ways for authentication. The simplest approach is to configure the server with a list of user’s public keys. The server can then generate a challenge for each connection request from a client, and the client must respond to the challenge by signing it with its private NKey, configured in the `nkey_file` or `nkey` field. For more details, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth). ### [](#user-credentials)User credentials NATS server also supports decentralized authentication based on JSON Web Tokens (JWTs). When a server is configured to use this authentication scheme, clients need a [user JWT](https://docs.nats.io/nats-server/configuration/securing_nats/jwt#json-web-tokens) and a corresponding [NKey secret](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth) to connect. You can use either of the following methods to supply the user JWT and NKey secret: - In the `user_credentials_file` field, enter the path to a file containing both the private key and the JWT. You can generate the file using the [nsc tool](https://docs.nats.io/nats-tools/nsc). - In the `user_jwt` field, enter a plain text JWT, and in the `user_nkey_seed` field, enter the plain text NKey seed or private key. For more details about authentication using JWTs, see the [NATS documentation](https://docs.nats.io/using-nats/developer/connecting/creds). ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed or private key for NATS authentication. NKeys provide secure, cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the corresponding user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#headers)`headers` Explicit message headers to add to messages. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). Requires version 4.1.0 or later. **Type**: `string` **Default**: `{}` ```yaml # Examples: headers: Content-Type: application/json Timestamp: ${!meta("Timestamp")} ``` ### [](#inject_tracing_map)`inject_tracing_map` EXPERIMENTAL: A [Bloblang mapping](../../../guides/bloblang/about/) used to inject an object containing tracing propagation information into outbound messages. The specification of the injected fields will match the format used by the service wide tracer. Requires version 4.23.0 or later. **Type**: `string` ```yaml # Examples: inject_tracing_map: meta = @.merge(this) # --- inject_tracing_map: root.meta.span = this ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `1024` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. **Type**: `int` ### [](#metadata)`metadata` Determine which (if any) metadata values should be added to messages as headers. **Type**: `object` ### [](#metadata-include_patterns)`metadata.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#metadata-include_prefixes)`metadata.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#subject)`subject` A subject to write to. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: subject: foo.bar.baz # --- subject: ${! meta("kafka_topic") } # --- subject: foo.${! json("meta.type") } ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. Requires version 4.60.0 or later. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 164: nats_kv **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/nats_kv.md --- # nats\_kv --- title: nats_kv latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/nats_kv page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/nats_kv.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/nats_kv.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/nats_kv/)[Cache](/redpanda-connect/components/caches/nats_kv/)[Input](/redpanda-connect/components/inputs/nats_kv/)[Processor](/redpanda-connect/components/processors/nats_kv/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/nats_kv/ "View the Cloud version of this component"), Self-Managed Put messages into a NATS key-value bucket. Introduced in version 4.12.0. #### Common ```yml outputs: label: "" nats_kv: urls: [] # No default (required) bucket: "" # No default (required) key: "" # No default (required) max_in_flight: 1024 ``` #### Advanced ```yml outputs: label: "" nats_kv: urls: [] # No default (required) max_reconnects: "" # No default (optional) bucket: "" # No default (required) key: "" # No default (required) max_in_flight: 1024 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) ``` The `key` field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries), which lets you create a unique key for each message. ## [](#connection-name)Connection name When monitoring and managing a production [NATS system](https://docs.nats.io/nats-concepts/overview), it is often useful to know which connection a message was sent or received from. To achieve this, set the connection name option when creating a NATS connection. Redpanda Connect can then automatically set the connection name to the NATS component label, so that monitoring tools between NATS and Redpanda Connect can stay in sync. ## [](#authentication)Authentication A number of Redpanda Connect components use NATS services. Each of these components support optional, advanced authentication parameters for [NKeys](https://docs.nats.io/nats-server/configuration/securing_nats/auth_intro/nkey_auth) and [user credentials](https://docs.nats.io/using-nats/developer/connecting/creds). For an in-depth guide, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/nats_admin/security/jwt). ### [](#nkeys)NKeys NATS server can use NKeys in several ways for authentication. The simplest approach is to configure the server with a list of user’s public keys. The server can then generate a challenge for each connection request from a client, and the client must respond to the challenge by signing it with its private NKey, configured in the `nkey_file` or `nkey` field. For more details, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth). ### [](#user-credentials)User credentials NATS server also supports decentralized authentication based on JSON Web Tokens (JWTs). When a server is configured to use this authentication scheme, clients need a [user JWT](https://docs.nats.io/nats-server/configuration/securing_nats/jwt#json-web-tokens) and a corresponding [NKey secret](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth) to connect. You can use either of the following methods to supply the user JWT and NKey secret: - In the `user_credentials_file` field, enter the path to a file containing both the private key and the JWT. You can generate the file using the [nsc tool](https://docs.nats.io/nats-tools/nsc). - In the `user_jwt` field, enter a plain text JWT, and in the `user_nkey_seed` field, enter the plain text NKey seed or private key. For more details about authentication using JWTs, see the [NATS documentation](https://docs.nats.io/using-nats/developer/connecting/creds). ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed or private key for NATS authentication. NKeys provide secure, cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the corresponding user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#bucket)`bucket` The name of the KV bucket. **Type**: `string` ```yaml # Examples: bucket: my_kv_bucket ``` ### [](#key)`key` The key for each message. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: key: foo # --- key: foo.bar.baz # --- key: foo.${! json("meta.type") } ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `1024` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. **Type**: `int` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. Requires version 4.60.0 or later. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 165: nats_stream **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/nats_stream.md --- # nats\_stream --- title: nats_stream latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/nats_stream page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/nats_stream.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/nats_stream.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/nats_stream/)[Input](/redpanda-connect/components/inputs/nats_stream/) **Available in:** Self-Managed Publish messages to a NATS Stream subject. #### Common ```yml outputs: label: "" nats_stream: urls: [] # No default (required) cluster_id: "" # No default (required) subject: "" # No default (required) client_id: "" max_in_flight: 64 ``` #### Advanced ```yml outputs: label: "" nats_stream: urls: [] # No default (required) max_reconnects: "" # No default (optional) cluster_id: "" # No default (required) subject: "" # No default (required) client_id: "" max_in_flight: 64 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) inject_tracing_map: "" # No default (optional) ``` > ⚠️ **CAUTION: Deprecation notice** > > Deprecation notice > > The NATS Streaming Server is being deprecated. Critical bug fixes and security fixes will be applied until June of 2023. NATS-enabled applications requiring persistence should use [JetStream](https://docs.nats.io/nats-concepts/jetstream). ## [](#authentication)Authentication There are a number of Redpanda Connect components that use NATS services. Each of these components support optional, advanced authentication parameters for [NKeys](https://docs.nats.io/nats-server/configuration/securing_nats/auth_intro/nkey_auth) and [user credentials](https://docs.nats.io/using-nats/developer/connecting/creds). For an in-depth guide, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/nats_admin/security/jwt). ### [](#nkeys)NKeys NATS server can use NKeys in several ways for authentication. The simplest approach is to configure the server with a list of user’s public keys. The server can then generate a challenge for each connection request from a client, and the client must respond to the challenge by signing it with its private NKey, configured in the `nkey_file` or `nkey` field. For more details, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth). ### [](#user-credentials)User credentials NATS server also supports decentralized authentication based on JSON Web Tokens (JWTs). When a server is configured to use this authentication scheme, clients need a [user JWT](https://docs.nats.io/nats-server/configuration/securing_nats/jwt#json-web-tokens) and a corresponding [NKey secret](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth) to connect. You can use either of the following methods to supply the user JWT and NKey secret: - In the `user_credentials_file` field, enter the path to a file containing both the private key and the JWT. You can generate the file using the [nsc tool](https://docs.nats.io/nats-tools/nsc). - In the `user_jwt` field, enter a plain text JWT, and in the `user_nkey_seed` field, enter the plain text NKey seed or private key. For more details about authentication using JWTs, see the [NATS documentation](https://docs.nats.io/using-nats/developer/connecting/creds). ## [](#performance)Performance Send multiple messages in flight in parallel for improved performance. You can tune the number of in flight messages (or message batches) with the field `max_in_flight`. ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed or private key for NATS authentication. NKeys provide secure, cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the corresponding user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#client_id)`client_id` The client ID to connect with. **Type**: `string` **Default**: `""` ### [](#cluster_id)`cluster_id` The cluster ID to publish to. **Type**: `string` ### [](#inject_tracing_map)`inject_tracing_map` EXPERIMENTAL: A [Bloblang mapping](../../../guides/bloblang/about/) used to inject an object containing tracing propagation information into outbound messages. The specification of the injected fields will match the format used by the service wide tracer. Requires version 4.23.0 or later. **Type**: `string` ```yaml # Examples: inject_tracing_map: meta = @.merge(this) # --- inject_tracing_map: root.meta.span = this ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. **Type**: `int` ### [](#subject)`subject` The subject to publish to. **Type**: `string` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. Requires version 4.60.0 or later. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 166: nats **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/nats.md --- # nats --- title: nats latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/nats page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/nats.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/nats.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/nats/)[Input](/redpanda-connect/components/inputs/nats/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/nats/ "View the Cloud version of this component"), Self-Managed Publish to an NATS subject. #### Common ```yml outputs: label: "" nats: urls: [] # No default (required) subject: "" # No default (required) headers: {} metadata: include_prefixes: [] include_patterns: [] max_in_flight: 64 ``` #### Advanced ```yml outputs: label: "" nats: urls: [] # No default (required) max_reconnects: "" # No default (optional) subject: "" # No default (required) headers: {} metadata: include_prefixes: [] include_patterns: [] max_in_flight: 64 tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) inject_tracing_map: "" # No default (optional) ``` This output interpolates functions within the subject field. For a full list of functions, see [configuration:interpolation.adoc#bloblang-queries](../../../configuration/interpolation/#bloblang-queries). ## [](#connection-name)Connection name When monitoring and managing a production [NATS system](https://docs.nats.io/nats-concepts/overview), it is often useful to know which connection a message was sent or received from. To achieve this, set the connection name option when creating a NATS connection. Redpanda Connect can then automatically set the connection name to the NATS component label, so that monitoring tools between NATS and Redpanda Connect can stay in sync. ## [](#authentication)Authentication A number of Redpanda Connect components use NATS services. Each of these components support optional, advanced authentication parameters for [NKeys](https://docs.nats.io/nats-server/configuration/securing_nats/auth_intro/nkey_auth) and [user credentials](https://docs.nats.io/using-nats/developer/connecting/creds). For an in-depth guide, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/nats_admin/security/jwt). ### [](#nkeys)NKeys NATS server can use NKeys in several ways for authentication. The simplest approach is to configure the server with a list of user’s public keys. The server can then generate a challenge for each connection request from a client, and the client must respond to the challenge by signing it with its private NKey, configured in the `nkey_file` or `nkey` field. For more details, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth). ### [](#user-credentials)User credentials NATS server also supports decentralized authentication based on JSON Web Tokens (JWTs). When a server is configured to use this authentication scheme, clients need a [user JWT](https://docs.nats.io/nats-server/configuration/securing_nats/jwt#json-web-tokens) and a corresponding [NKey secret](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth) to connect. You can use either of the following methods to supply the user JWT and NKey secret: - In the `user_credentials_file` field, enter the path to a file containing both the private key and the JWT. You can generate the file using the [nsc tool](https://docs.nats.io/nats-tools/nsc). - In the `user_jwt` field, enter a plain text JWT, and in the `user_nkey_seed` field, enter the plain text NKey seed or private key. For more details about authentication using JWTs, see the [NATS documentation](https://docs.nats.io/using-nats/developer/connecting/creds). ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed or private key for NATS authentication. NKeys provide secure, cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the corresponding user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#headers)`headers` Explicit message headers to add to messages. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `{}` ```yaml # Examples: headers: Content-Type: application/json Timestamp: ${!meta("Timestamp")} ``` ### [](#inject_tracing_map)`inject_tracing_map` EXPERIMENTAL: A [Bloblang mapping](../../../guides/bloblang/about/) used to inject an object containing tracing propagation information into outbound messages. The specification of the injected fields will match the format used by the service wide tracer. Requires version 4.23.0 or later. **Type**: `string` ```yaml # Examples: inject_tracing_map: meta = @.merge(this) # --- inject_tracing_map: root.meta.span = this ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. **Type**: `int` ### [](#metadata)`metadata` Determine which (if any) metadata values should be added to messages as headers. **Type**: `object` ### [](#metadata-include_patterns)`metadata.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#metadata-include_prefixes)`metadata.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#subject)`subject` The subject to publish to. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: subject: foo.bar.baz ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. Requires version 4.60.0 or later. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 167: nsq **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/nsq.md --- # nsq --- title: nsq latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/nsq page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/nsq.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/nsq.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/nsq/)[Input](/redpanda-connect/components/inputs/nsq/) **Available in:** Self-Managed Publish to an NSQ topic. #### Common ```yml outputs: label: "" nsq: nsqd_tcp_address: "" # No default (required) topic: "" # No default (required) user_agent: "" # No default (optional) max_in_flight: 64 ``` #### Advanced ```yml outputs: label: "" nsq: nsqd_tcp_address: "" # No default (required) topic: "" # No default (required) user_agent: "" # No default (optional) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] max_in_flight: 64 ``` The `topic` field can be dynamically set using function interpolations described [here](../../../configuration/interpolation/#bloblang-queries). When sending batched messages these interpolations are performed per message part. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ## [](#fields)Fields ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#nsqd_tcp_address)`nsqd_tcp_address` The address of the target NSQD server. **Type**: `string` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#topic)`topic` The topic to publish to. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#user_agent)`user_agent` A user agent to assume when connecting. **Type**: `string` --- # Page 168: ockam_kafka **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/ockam_kafka.md --- # ockam\_kafka --- title: ockam_kafka latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/ockam_kafka page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/ockam_kafka.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/ockam_kafka.adoc page-git-created-date: "2024-11-25" page-git-modified-date: "2025-10-06" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/ockam_kafka/)[Input](/redpanda-connect/components/inputs/ockam_kafka/) **Available in:** Self-Managed Uses [Ockam](https://docs.ockam.io/) to encrypt and write end-to-end encrypted messages to a Kafka topic. You can read encrypted messages using the `ockam_kafka` input or by [creating a Kafka Portal Inlet](https://command.ockam.io/manual/ockam-kafka-inlet-create.html) using Ockam Command. [Ockam Secure Channels](https://docs.ockam.io/reference/command/secure-channels) guarantee that consumers of topics only receive data from authenticated and authorized producers, and that the data has not been leaked or tampered with in-flight. Neither Kafka brokers, service providers nor other components can see or manipulate the messages. > 📝 **NOTE** > > You can use Ockam to encrypt whole messages, or specific fields in a message. Introduced in version 4.33.0. #### Common ```yml outputs: label: "" ockam_kafka: kafka: seed_brokers: [] # No default (optional) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] max_in_flight: 10 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) partitioner: "" # No default (optional) idempotent_write: true compression: "" # No default (optional) allow_auto_topic_creation: true timeout: 10s max_message_bytes: 1MiB broker_write_max_bytes: 100MiB topic: "" # No default (required) key: "" # No default (optional) partition: "" # No default (optional) metadata: include_prefixes: [] include_patterns: [] timestamp_ms: "" # No default (optional) disable_content_encryption: false enrollment_ticket: "" # No default (optional) identity_name: "" # No default (optional) allow: self route_to_kafka_outlet: self allow_consumer: self route_to_consumer: /ip4/127.0.0.1/tcp/6262 encrypted_fields: [] ``` #### Advanced ```yml outputs: label: "" ockam_kafka: kafka: seed_brokers: [] # No default (optional) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] max_in_flight: 10 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) partitioner: "" # No default (optional) idempotent_write: true compression: "" # No default (optional) allow_auto_topic_creation: true timeout: 10s max_message_bytes: 1MiB broker_write_max_bytes: 100MiB topic: "" # No default (required) key: "" # No default (optional) partition: "" # No default (optional) metadata: include_prefixes: [] include_patterns: [] timestamp_ms: "" # No default (optional) disable_content_encryption: false enrollment_ticket: "" # No default (optional) identity_name: "" # No default (optional) allow: self route_to_kafka_outlet: self allow_consumer: self route_to_consumer: /ip4/127.0.0.1/tcp/6262 encrypted_fields: [] ``` ## [](#fields)Fields ### [](#allow)`allow` Use in conjunction with the `route_to_kafka_outlet` field to specify an access control policy for the Kafka Portal Outlet. For example, setting this value to `kafka_us_east` forces the Kafka Outlet to present an Ockam credential, which confirms that the Outlet has the attribute `kafka_us_east=true`. **Type**: `string` **Default**: `self` ### [](#allow_consumer)`allow_consumer` Specify an access control policy for consumers. For example, setting this value to `orders_consumer` forces the consumer to present an Ockam credential, which confirms that the consumer has the attribute `orders_consumer=true`. **Type**: `string` **Default**: `self` ### [](#disable_content_encryption)`disable_content_encryption` If this value is set to `true`: - Only message payloads remain unencrypted. This setting does not disable TLS or any other transport-layer encryption that may also be enabled. - All other `ockam_kafka` inlets and outlets must also have their settings set to `true`. **Type**: `bool` **Default**: `false` ### [](#encrypted_fields)`encrypted_fields[]` The fields to encrypt in the kafka messages, assuming the record is a valid JSON map. By default, the whole record is encrypted. **Type**: `array` **Default**: `[]` ### [](#enrollment_ticket)`enrollment_ticket` The path to a file or a URL where the enrollment ticket value is stored, or an inline hex-encoded value of the enrollment ticket (optional). You can generate a new ticket using the [`ockam project ticket` command](https://command.ockam.io/manual/ockam-project-ticket.html). **Type**: `string` ### [](#identity_name)`identity_name` The name of the [Ockam identity](https://command.ockam.io/manual/ockam-identity.html) to use. If this value is not provided, the default Ockam identity is automatically generated and used (optional). **Type**: `string` ### [](#kafka)`kafka` **Type**: `object` ### [](#kafka-allow_auto_topic_creation)`kafka.allow_auto_topic_creation` Enables topics to be auto created if they do not exist when fetching their metadata. **Type**: `bool` **Default**: `true` ### [](#kafka-batching)`kafka.batching` Configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#kafka-batching-byte_size)`kafka.batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#kafka-batching-check)`kafka.batching.check` A [Bloblang query](../../../guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#kafka-batching-count)`kafka.batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#kafka-batching-period)`kafka.batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#kafka-batching-processors)`kafka.batching.processors[]` For aggregating and archiving message batches, you can add a list of [processors](../../processors/about/) to apply to a batch as it is flushed (optional). All resulting messages are flushed as a single batch even when you configure processors to split the batch into smaller batches. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#kafka-broker_write_max_bytes)`kafka.broker_write_max_bytes` The maximum number of bytes this output can write to a broker connection in a single write. This field corresponds to Kafka’s `socket.request.max.bytes`. **Type**: `string` **Default**: `100MiB` ```yaml # Examples: broker_write_max_bytes: 128MB # --- broker_write_max_bytes: 50mib ``` ### [](#kafka-compression)`kafka.compression` Set an explicit compression type (optional). The preferred default is `snappy` when the broker supports it, with a fall back to `none`. **Type**: `string` **Options**: `lz4`, `snappy`, `gzip`, `none`, `zstd` ### [](#kafka-idempotent_write)`kafka.idempotent_write` Enables the `idempotent_write` producer option. This requires `IDEMPOTENT_WRITE` permission on `CLUSTER`. Disable this option if the correct permission is not available. **Type**: `bool` **Default**: `true` ### [](#kafka-key)`kafka.key` Populates a key for each message (optional). This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#kafka-max_in_flight)`kafka.max_in_flight` The maximum number of message batches to send in parallel at any given time. **Type**: `int` **Default**: `10` ### [](#kafka-max_message_bytes)`kafka.max_message_bytes` The maximum size of an individual message in bytes. Messages larger than this value are rejected. This field is equivalent to Kafka’s `max.message.bytes`. **Type**: `string` **Default**: `1MiB` ```yaml # Examples: max_message_bytes: 100MB # --- max_message_bytes: 50mib ``` ### [](#kafka-metadata)`kafka.metadata` Determines which metadata values are added to messages as headers. **Type**: `object` ### [](#kafka-metadata-include_patterns)`kafka.metadata.include_patterns[]` A list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#kafka-metadata-include_prefixes)`kafka.metadata.include_prefixes[]` A list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#kafka-partition)`kafka.partition` Set an explicit partition for each message (optional). To use this field, set the `partitioner` to `manual`. You must provide an interpolation string that is a valid integer. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: partition: ${! meta("partition") } ``` ### [](#kafka-partitioner)`kafka.partitioner` Override the default murmur2 hashing partitioner (optional). **Type**: `string` | Option | Summary | | --- | --- | | least_backup | Chooses the least backed up partition (the partition with the fewest amount of buffered records). Partitions are selected per batch. | | manual | Manually select a partition for each message, requires the field partition to be specified. | | murmur2_hash | Kafka’s default hash algorithm that uses a 32-bit murmur2 hash of the key to compute which partition the record will be on. | | round_robin | Round-robin’s messages through all available partitions. This algorithm has lower throughput and causes higher CPU load on brokers, but can be useful if you want to ensure an even distribution of records to partitions. | ### [](#kafka-seed_brokers)`kafka.seed_brokers[]` A list of broker addresses to connect to (optional). List items that contain commas are expanded into multiple addresses. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" # --- seed_brokers: - "foo:9092" - "bar:9092" # --- seed_brokers: - "foo:9092,bar:9092" ``` ### [](#kafka-timeout)`kafka.timeout` The maximum period of time allowed for sending messages before a request is abandoned and a retry attempted. **Type**: `string` **Default**: `10s` ### [](#kafka-timestamp_ms)`kafka.timestamp_ms` Set a timestamp (in milliseconds) for each message (optional). Leave this field empty to use the current timestamp. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: timestamp_ms: ${! timestamp_unix_milli() } # --- timestamp_ms: ${! metadata("kafka_timestamp_ms") } ``` ### [](#kafka-tls)`kafka.tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#kafka-tls-client_certs)`kafka.tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#kafka-tls-client_certs-cert)`kafka.tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#kafka-tls-client_certs-cert_file)`kafka.tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#kafka-tls-client_certs-key)`kafka.tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#kafka-tls-client_certs-key_file)`kafka.tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#kafka-tls-client_certs-password)`kafka.tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#kafka-tls-enable_renegotiation)`kafka.tls.enable_renegotiation` Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#kafka-tls-enabled)`kafka.tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#kafka-tls-root_cas)`kafka.tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#kafka-tls-root_cas_file)`kafka.tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#kafka-tls-skip_cert_verify)`kafka.tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#kafka-topic)`kafka.topic` The Kafka topic to write messages to (required). This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#route_to_consumer)`route_to_consumer` The route to the Kafka consumer. For example, `/project/default/service/forward_to_orders_consumer/secure/api` would connect to a consumer exposed through a relay named `orders_consumer`. **Type**: `string` **Default**: `/ip4/127.0.0.1/tcp/6262` ### [](#route_to_kafka_outlet)`route_to_kafka_outlet` The route to reach the Kafka Outlet of your Ockam portal. For example, `/project/default`. **Type**: `string` **Default**: `self` --- # Page 169: opensearch **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/opensearch.md --- # opensearch --- title: opensearch latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/opensearch page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/opensearch.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/opensearch.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/opensearch/ "View the Cloud version of this component"), Self-Managed Publishes messages into an Elasticsearch index. If the index does not exist then it is created with a dynamic mapping. #### Common ```yml outputs: label: "" opensearch: urls: [] # No default (required) index: "" # No default (required) action: "" # No default (required) id: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" opensearch: urls: [] # No default (required) index: "" # No default (required) action: "" # No default (required) id: "" # No default (required) pipeline: "" routing: "" tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] max_in_flight: 64 basic_auth: enabled: false username: "" password: "" batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) aws: enabled: false region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` Both the `id` and `index` fields can be dynamically set using function interpolations described [here](../../../configuration/interpolation/#bloblang-queries). When sending batched messages these interpolations are performed per message part. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](../../../configuration/batching/). ## [](#examples)Examples ### [](#updating-documents)Updating Documents When [updating documents](https://opensearch.org/docs/latest/api-reference/document-apis/update-document/) the request body should contain a combination of a `doc`, `upsert`, and/or `script` fields at the top level, this should be done via mapping processors. ```yaml output: processors: - mapping: | meta id = this.id root.doc = this opensearch: urls: [ TODO ] index: foo id: ${! @id } action: update ``` ## [](#fields)Fields ### [](#action)`action` The action to take on the document. This field must resolve to one of the following action types: `index`, `update` or `delete`. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#aws)`aws` Enables and customises connectivity to Amazon Elastic Service. **Type**: `object` ### [](#aws-credentials)`aws.credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](../../../guides/cloud/aws/). **Type**: `object` ### [](#aws-credentials-from_ec2_role)`aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#aws-credentials-id)`aws.credentials.id` The ID of credentials to use. **Type**: `string` ### [](#aws-credentials-profile)`aws.credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#aws-credentials-role)`aws.credentials.role` A role ARN to assume. **Type**: `string` ### [](#aws-credentials-role_external_id)`aws.credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#aws-credentials-secret)`aws.credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#aws-credentials-token)`aws.credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#aws-enabled)`aws.enabled` Whether to connect to Amazon Elastic Service. **Type**: `bool` **Default**: `false` ### [](#aws-endpoint)`aws.endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#aws-region)`aws.region` The AWS region to target. **Type**: `string` ### [](#aws-tcp)`aws.tcp` TCP socket configuration. **Type**: `object` ### [](#aws-tcp-connect_timeout)`aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#aws-tcp-keep_alive)`aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#aws-tcp-keep_alive-count)`aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#aws-tcp-keep_alive-idle)`aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#aws-tcp-keep_alive-interval)`aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#aws-tcp-tcp_user_timeout)`aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#basic_auth)`basic_auth` Allows you to specify basic authentication. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#id)`id` The ID for indexed messages. Interpolation should be used in order to create a unique ID for each message. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: id: ${!counter()}-${!timestamp_unix()} ``` ### [](#index)`index` The index to place messages. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#pipeline)`pipeline` An optional pipeline id to preprocess incoming documents. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#routing)`routing` The routing key to use for the document. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If an item of the list contains commas it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "http://localhost:9200" ``` --- # Page 170: otlp_grpc **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/otlp_grpc.md --- # otlp\_grpc --- title: otlp_grpc latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/otlp_grpc page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/otlp_grpc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/otlp_grpc.adoc description: Send OpenTelemetry traces, logs, and metrics via OTLP/gRPC protocol. page-git-created-date: "2026-01-23" page-git-modified-date: "2026-01-23" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/otlp_grpc/)[Input](/redpanda-connect/components/inputs/otlp_grpc/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/otlp_grpc/ "View the Cloud version of this component"), Self-Managed **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Send OpenTelemetry traces, logs, and metrics via OTLP/gRPC protocol. Introduced in version 4.78.0. Sends OpenTelemetry telemetry data to a remote collector via OTLP/gRPC protocol. Accepts batches of Redpanda OTEL v1 protobuf messages (spans, log records, or metrics) and converts them to OTLP format for transmission to OpenTelemetry collectors. #### Common ```yml outputs: label: "" otlp_grpc: endpoint: "" # No default (required) max_in_flight: 64 ``` #### Advanced ```yml outputs: label: "" otlp_grpc: endpoint: "" # No default (required) headers: {} timeout: 30s compression: gzip tls: enabled: false skip_cert_verify: false cert_file: "" key_file: "" tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s oauth2: enabled: false client_key: "" client_secret: "" token_url: "" scopes: [] endpoint_params: {} max_in_flight: 64 ``` ## [](#input-format)Input format Expects messages in Redpanda OTEL v1 protobuf format with metadata: - `signal_type`: "trace", "log", or "metric" Each batch must contain messages of the same signal type. The entire batch is converted to a single OTLP export request and sent via gRPC. ## [](#authentication)Authentication Supports multiple authentication methods: - Bearer token authentication (via `auth_token` field) - OAuth v2 (via `oauth2` configuration block) > 📝 **NOTE** > > OAuth2 requires TLS to be enabled. ## [](#fields)Fields ### [](#compression)`compression` Compression type for gRPC requests. Options: 'gzip' or 'none'. **Type**: `string` **Default**: `gzip` **Options**: `gzip`, `none` ### [](#endpoint)`endpoint` The gRPC endpoint of the remote OTLP collector. **Type**: `string` ### [](#headers)`headers` A map of headers to add to the gRPC request metadata. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `{}` ```yaml # Examples: headers: X-Custom-Header: value traceparent: ${! tracing_span().traceparent } ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#oauth2)`oauth2` Allows you to specify open authentication via OAuth version 2 using the client credentials token flow. **Type**: `object` ### [](#oauth2-client_key)`oauth2.client_key` A value used to identify the client to the token provider. **Type**: `string` **Default**: `""` ### [](#oauth2-client_secret)`oauth2.client_secret` A secret used to establish ownership of the client key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth2-enabled)`oauth2.enabled` Whether to use OAuth version 2 in requests. **Type**: `bool` **Default**: `false` ### [](#oauth2-endpoint_params)`oauth2.endpoint_params` A list of optional endpoint parameters, values should be arrays of strings. **Type**: `object` **Default**: `{}` ```yaml # Examples: endpoint_params: audience: - https://example.com resource: - https://api.example.com ``` ### [](#oauth2-scopes)`oauth2.scopes[]` A list of optional requested permissions. **Type**: `array` **Default**: `[]` ### [](#oauth2-token_url)`oauth2.token_url` The URL of the token provider. **Type**: `string` **Default**: `""` ### [](#tcp)`tcp` TCP socket configuration. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` Timeout for gRPC requests. **Type**: `string` **Default**: `30s` ### [](#tls)`tls` TLS configuration for gRPC client. **Type**: `object` ### [](#tls-cert_file)`tls.cert_file` Path to the TLS certificate file for client authentication. **Type**: `string` **Default**: `""` ### [](#tls-enabled)`tls.enabled` Enable TLS connections. **Type**: `bool` **Default**: `false` ### [](#tls-key_file)`tls.key_file` Path to the TLS key file for client authentication. **Type**: `string` **Default**: `""` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Skip certificate verification (insecure). **Type**: `bool` **Default**: `false` --- # Page 171: otlp_http **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/otlp_http.md --- # otlp\_http --- title: otlp_http latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/otlp_http page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/otlp_http.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/otlp_http.adoc description: Send OpenTelemetry traces, logs, and metrics via OTLP/HTTP protocol. page-git-created-date: "2026-01-23" page-git-modified-date: "2026-01-23" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/otlp_http/)[Input](/redpanda-connect/components/inputs/otlp_http/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/otlp_http/ "View the Cloud version of this component"), Self-Managed **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Send OpenTelemetry traces, logs, and metrics via OTLP/HTTP protocol. Introduced in version 4.78.0. Sends OpenTelemetry telemetry data to a remote collector via OTLP/HTTP protocol. Accepts batches of Redpanda OTEL v1 protobuf messages (spans, log records, or metrics) and converts them to OTLP format for transmission to OpenTelemetry collectors. #### Common ```yml outputs: label: "" otlp_http: endpoint: "" # No default (required) max_in_flight: 64 ``` #### Advanced ```yml outputs: label: "" otlp_http: endpoint: "" # No default (required) content_type: protobuf headers: {} timeout: 30s proxy_url: "" follow_redirects: false disable_http2: false tls: enabled: false skip_cert_verify: false cert_file: "" key_file: "" tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} oauth2: enabled: false client_key: "" client_secret: "" token_url: "" scopes: [] endpoint_params: {} max_in_flight: 64 ``` ## [](#input-format)Input format Expects messages in Redpanda OTEL v1 protobuf format with metadata: - `signal_type`: "trace", "log", or "metric" Each batch must contain messages of the same signal type. The entire batch is converted to a single OTLP export request and sent via HTTP POST. ## [](#endpoints)Endpoints The output automatically appends the signal type path to the base endpoint: - Traces: `{endpoint}/v1/traces` - Logs: `{endpoint}/v1/logs` - Metrics: `{endpoint}/v1/metrics` ## [](#content-types)Content types Supports two content types: - `protobuf` (default): `application/x-protobuf` - `json`: `application/json` ## [](#authentication)Authentication Supports multiple authentication methods: - Basic authentication - OAuth v1 - OAuth v2 - JWT ## [](#fields)Fields ### [](#basic_auth)`basic_auth` Allows you to specify basic authentication. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#content_type)`content_type` Content type for HTTP requests. Options: 'protobuf' or 'json'. **Type**: `string` **Default**: `protobuf` **Options**: `protobuf`, `json` ### [](#disable_http2)`disable_http2` Whether or not to disable HTTP/2. **Type**: `bool` **Default**: `false` ### [](#endpoint)`endpoint` The HTTP endpoint of the remote OTLP collector (without the signal path). **Type**: `string` ### [](#follow_redirects)`follow_redirects` Transparently follow redirects, i.e. responses with 300-399 status codes. If disabled, the response message will contain the body, status, and headers from the redirect response and the processor will not make a request to the URL set in the Location header of the response. **Type**: `bool` **Default**: `false` ### [](#headers)`headers` A map of headers to add to the request. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `{}` ```yaml # Examples: headers: X-Custom-Header: value traceparent: ${! tracing_span().traceparent } ``` ### [](#jwt)`jwt` Beta Allows you to specify JWT authentication. **Type**: `object` ### [](#jwt-claims)`jwt.claims` A value used to identify the claims that issued the JWT. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` Add optional key/value headers to the JWT. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` A file with the PEM encoded via PKCS1 or PKCS8 as private key. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` A method used to sign the token such as RS256, RS384, RS512 or EdDSA. **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#oauth)`oauth` Allows you to specify open authentication via OAuth version 1. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` A value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` A secret provided in order to establish ownership of a given access token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` A value used to identify the client to the service provider. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` A secret used to establish ownership of the consumer key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#oauth2)`oauth2` Allows you to specify open authentication via OAuth version 2 using the client credentials token flow. **Type**: `object` ### [](#oauth2-client_key)`oauth2.client_key` A value used to identify the client to the token provider. **Type**: `string` **Default**: `""` ### [](#oauth2-client_secret)`oauth2.client_secret` A secret used to establish ownership of the client key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth2-enabled)`oauth2.enabled` Whether to use OAuth version 2 in requests. **Type**: `bool` **Default**: `false` ### [](#oauth2-endpoint_params)`oauth2.endpoint_params` A list of optional endpoint parameters, values should be arrays of strings. **Type**: `object` **Default**: `{}` ```yaml # Examples: endpoint_params: audience: - https://example.com resource: - https://api.example.com ``` ### [](#oauth2-scopes)`oauth2.scopes[]` A list of optional requested permissions. **Type**: `array` **Default**: `[]` ### [](#oauth2-token_url)`oauth2.token_url` The URL of the token provider. **Type**: `string` **Default**: `""` ### [](#proxy_url)`proxy_url` An optional HTTP proxy URL. **Type**: `string` **Default**: `""` ### [](#tcp)`tcp` TCP socket configuration. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` Timeout for HTTP requests. **Type**: `string` **Default**: `30s` ### [](#tls)`tls` TLS configuration for HTTP client. **Type**: `object` ### [](#tls-cert_file)`tls.cert_file` Path to the TLS certificate file for client authentication. **Type**: `string` **Default**: `""` ### [](#tls-enabled)`tls.enabled` Enable TLS connections. **Type**: `bool` **Default**: `false` ### [](#tls-key_file)`tls.key_file` Path to the TLS key file for client authentication. **Type**: `string` **Default**: `""` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Skip certificate verification (insecure). **Type**: `bool` **Default**: `false` --- # Page 172: pinecone **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/pinecone.md --- # pinecone --- title: pinecone latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/pinecone page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/pinecone.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/pinecone.adoc page-git-created-date: "2024-08-15" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/pinecone/ "View the Cloud version of this component"), Self-Managed Inserts items into a Pinecone index. Introduced in version 4.31.0. #### Common ```yml outputs: label: "" pinecone: max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) host: "" # No default (required) api_key: "" # No default (required) operation: upsert-vectors id: "" # No default (required) vector_mapping: "" # No default (optional) metadata_mapping: "" # No default (optional) ``` #### Advanced ```yml outputs: label: "" pinecone: max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) host: "" # No default (required) api_key: "" # No default (required) operation: upsert-vectors namespace: "" id: "" # No default (required) vector_mapping: "" # No default (optional) metadata_mapping: "" # No default (optional) ``` ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](../../../configuration/batching/). ## [](#fields)Fields ### [](#api_key)`api_key` The Pinecone API key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#host)`host` The host for the Pinecone index. **Type**: `string` ### [](#id)`id` The ID for the index entry in Pinecone. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#metadata_mapping)`metadata_mapping` An optional mapping of message to metadata in the Pinecone index entry. **Type**: `string` ```yaml # Examples: metadata_mapping: root = @ # --- metadata_mapping: root = metadata() # --- metadata_mapping: root = {"summary": this.summary, "foo": this.other_field} ``` ### [](#namespace)`namespace` The namespace to write to - writes to the default namespace by default. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#operation)`operation` The operation to perform against the Pinecone index. **Type**: `string` **Default**: `upsert-vectors` **Options**: `update-vector`, `upsert-vectors`, `delete-vectors` ### [](#vector_mapping)`vector_mapping` The mapping to extract out the vector from the document. The result must be a floating point array. Required if not a delete operation. **Type**: `string` ```yaml # Examples: vector_mapping: root = this.embeddings_vector # --- vector_mapping: root = [1.2, 0.5, 0.76] ``` --- # Page 173: pulsar **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/pulsar.md --- # pulsar --- title: pulsar latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/pulsar page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/pulsar.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/pulsar.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/pulsar/)[Input](/redpanda-connect/components/inputs/pulsar/) **Available in:** Self-Managed Write messages to an Apache Pulsar server. Introduced in version 3.43.0. #### Common ```yml outputs: label: "" pulsar: url: "" # No default (required) topic: "" # No default (required) tls: root_cas_file: "" key: "" ordering_key: "" max_in_flight: 64 ``` #### Advanced ```yml outputs: label: "" pulsar: url: "" # No default (required) topic: "" # No default (required) tls: root_cas_file: "" key: "" ordering_key: "" max_in_flight: 64 auth: oauth2: enabled: false audience: "" issuer_url: "" scope: "" private_key_file: "" token: enabled: false token: "" ``` ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of Pulsar authentication methods. Requires version 3.60.0 or later. **Type**: `object` ### [](#auth-oauth2)`auth.oauth2` Parameters for Pulsar OAuth2 authentication. **Type**: `object` ### [](#auth-oauth2-audience)`auth.oauth2.audience` OAuth2 audience. **Type**: `string` **Default**: `""` ### [](#auth-oauth2-enabled)`auth.oauth2.enabled` Whether OAuth2 is enabled. **Type**: `bool` **Default**: `false` ### [](#auth-oauth2-issuer_url)`auth.oauth2.issuer_url` OAuth2 issuer URL. **Type**: `string` **Default**: `""` ### [](#auth-oauth2-private_key_file)`auth.oauth2.private_key_file` The path to a file containing a private key. **Type**: `string` **Default**: `""` ### [](#auth-oauth2-scope)`auth.oauth2.scope` OAuth2 scope to request. **Type**: `string` **Default**: `""` ### [](#auth-token)`auth.token` Parameters for Pulsar Token authentication. **Type**: `object` ### [](#auth-token-enabled)`auth.token.enabled` Whether Token Auth is enabled. **Type**: `bool` **Default**: `false` ### [](#auth-token-token)`auth.token.token` Actual base64 encoded token. **Type**: `string` **Default**: `""` ### [](#key)`key` An optional key to populate for each message. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#ordering_key)`ordering_key` The ordering key to publish messages with. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#tls)`tls` Specify the path to a custom CA certificate to trust broker TLS service. **Type**: `object` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#topic)`topic` The topic to publish to. **Type**: `string` ### [](#url)`url` A URL to connect to. **Type**: `string` ```yaml # Examples: url: pulsar://localhost:6650 # --- url: pulsar://pulsar.us-west.example.com:6650 # --- url: pulsar+ssl://pulsar.us-west.example.com:6651 ``` --- # Page 174: pusher **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/pusher.md --- # pusher --- title: pusher latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/pusher page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/pusher.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/pusher.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** Self-Managed Output for publishing messages to Pusher API ([https://pusher.com](https://pusher.com)) Introduced in version 4.3.0. #### Common ```yml outputs: label: "" pusher: batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) channel: "" # No default (required) event: "" # No default (required) appId: "" # No default (required) key: "" # No default (required) secret: "" # No default (required) cluster: "" # No default (required) secure: true max_in_flight: 1 ``` #### Advanced ```yml outputs: label: "" pusher: batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) channel: "" # No default (required) event: "" # No default (required) appId: "" # No default (required) key: "" # No default (required) secret: "" # No default (required) cluster: "" # No default (required) secure: true max_in_flight: 1 ``` ## [](#fields)Fields ### [](#appid)`appId` Pusher app id **Type**: `string` ### [](#batching)`batching` maximum batch size is 10 (limit of the pusher library) **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#channel)`channel` Pusher channel to publish to. Interpolation functions can also be used This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: channel: my_channel # --- channel: ${!json("id")} ``` ### [](#cluster)`cluster` Pusher cluster **Type**: `string` ### [](#event)`event` Event to publish to **Type**: `string` ### [](#key)`key` Pusher key **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of parallel message batches to have in flight at any given time. **Type**: `int` **Default**: `1` ### [](#secret)`secret` Pusher secret **Type**: `string` ### [](#secure)`secure` Enable SSL encryption **Type**: `bool` **Default**: `true` --- # Page 175: qdrant **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/qdrant.md --- # qdrant --- title: qdrant latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/qdrant page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/qdrant.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/qdrant.adoc page-git-created-date: "2024-08-15" page-git-modified-date: "2025-10-06" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/qdrant/)[Processor](/redpanda-connect/components/processors/qdrant/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/qdrant/ "View the Cloud version of this component"), Self-Managed Adds items to a [Qdrant](https://qdrant.tech/) collection Introduced in version 4.33.0. #### Common ```yml outputs: label: "" qdrant: max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) grpc_host: "" # No default (required) api_token: "" collection_name: "" # No default (required) id: "" # No default (required) vector_mapping: "" # No default (required) payload_mapping: root = {} ``` #### Advanced ```yml outputs: label: "" qdrant: max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) grpc_host: "" # No default (required) api_token: "" tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] collection_name: "" # No default (required) id: "" # No default (required) vector_mapping: "" # No default (required) payload_mapping: root = {} ``` ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](../../../configuration/batching/). ## [](#fields)Fields ### [](#api_token)`api_token` The Qdrant API token for authentication. Defaults to an empty string. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#collection_name)`collection_name` The name of the collection in Qdrant. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#grpc_host)`grpc_host` The gRPC host of the Qdrant server. **Type**: `string` ```yaml # Examples: grpc_host: localhost:6334 # --- grpc_host: xyz-example.eu-central.aws.cloud.qdrant.io:6334 ``` ### [](#id)`id` The ID of the point to insert. Can be a UUID string or positive integer. **Type**: `string` ```yaml # Examples: id: root = "dc88c126-679f-49f5-ab85-04b77e8c2791" # --- id: root = 832 ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#payload_mapping)`payload_mapping` An optional mapping of message to payload associated with the point. **Type**: `string` **Default**: `root = {}` ```yaml # Examples: payload_mapping: root = {"field": this.value, "field_2": 987} # --- payload_mapping: root = metadata() ``` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#vector_mapping)`vector_mapping` The mapping to extract the vector from the document. **Type**: `string` ```yaml # Examples: vector_mapping: root = {"dense_vector": [0.352,0.532,0.754],"sparse_vector": {"indices": [23,325,532],"values": [0.352,0.532,0.532]}, "multi_vector": [[0.352,0.532],[0.352,0.532]]} # --- vector_mapping: root = [1.2, 0.5, 0.76] # --- vector_mapping: root = this.vector # --- vector_mapping: root = [[0.352,0.532,0.532,0.234],[0.352,0.532,0.532,0.234]] # --- vector_mapping: root = {"some_sparse": {"indices":[23,325,532],"values":[0.352,0.532,0.532]}} # --- vector_mapping: root = {"some_multi": [[0.352,0.532,0.532,0.234],[0.352,0.532,0.532,0.234]]} # --- vector_mapping: root = {"some_dense": [0.352,0.532,0.532,0.234]} ``` --- # Page 176: questdb **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/questdb.md --- # questdb --- title: questdb latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/questdb page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/questdb.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/questdb.adoc page-git-created-date: "2024-10-16" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/questdb/ "View the Cloud version of this component"), Self-Managed Pushes messages to a [QuestDB](https://questdb.io/docs/) table. Introduced in version 4.37.0. #### Common ```yml outputs: label: "" questdb: max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) address: "" # No default (required) username: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) table: "" # No default (required) designated_timestamp_field: "" # No default (optional) designated_timestamp_unit: auto timestamp_string_fields: [] # No default (optional) timestamp_string_format: Jan _2 15:04:05.000000Z0700 symbols: [] # No default (optional) doubles: [] # No default (optional) error_on_empty_messages: false ``` #### Advanced ```yml outputs: label: "" questdb: max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] address: "" # No default (required) username: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) retry_timeout: "" # No default (optional) request_timeout: "" # No default (optional) request_min_throughput: "" # No default (optional) table: "" # No default (required) designated_timestamp_field: "" # No default (optional) designated_timestamp_unit: auto timestamp_string_fields: [] # No default (optional) timestamp_string_format: Jan _2 15:04:05.000000Z0700 symbols: [] # No default (optional) doubles: [] # No default (optional) error_on_empty_messages: false ``` > ❗ **IMPORTANT** > > Redpanda Data recommends enabling the dedupe feature on the QuestDB server. For more information about deploying, configuring, and using QuestDB, see the [QuestDB documentation](https://questdb.io/docs/). ## [](#performance)Performance For improved performance, this output sends multiple messages in parallel. You can tune the maximum number of in-flight messages (or message batches), using the `max_in_flight` field. You can configure batches at both the input and output level. For more information, see [Message Batching](../../../configuration/batching/). ## [](#fields)Fields ### [](#address)`address` The host and port of the QuestDB server. **Type**: `string` ```yaml # Examples: address: localhost:9000 ``` ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#designated_timestamp_field)`designated_timestamp_field` The name of the designated timestamp field in QuestDB. **Type**: `string` ### [](#designated_timestamp_unit)`designated_timestamp_unit` Units used for the designated timestamp field in QuestDB. **Type**: `string` **Default**: `auto` ### [](#doubles)`doubles[]` Columns that must be the `double` type, with `int` as the default. **Type**: `array` ### [](#error_on_empty_messages)`error_on_empty_messages` Mark a message as an error if it is empty after field validation. **Type**: `bool` **Default**: `false` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this value to improve throughput. **Type**: `int` **Default**: `64` ### [](#password)`password` The password to use for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#request_min_throughput)`request_min_throughput` The minimum expected throughput in bytes per second for HTTP requests. If the throughput is lower than this value, the connection times out. The `quest_db` output uses this value to calculate an additional timeout on top of the `request_timeout`. This setting is useful for large requests. Set it to `0` to disable this logic. **Type**: `int` ### [](#request_timeout)`request_timeout` The period of time to wait for a response from the QuestDB server in addition to any connection timeout calculated for the `request_min_throughput` field. **Type**: `string` ### [](#retry_timeout)`retry_timeout` The period of time to continue retrying after a failed HTTP request. The interval between retries is an exponential backoff starting at 10 ms, and doubling after each failed attempt up to a maximum of 1 second. **Type**: `string` ### [](#symbols)`symbols[]` Columns that must be the `symbol` type. String values default to `string` types. **Type**: `array` ### [](#table)`table` The destination table in QuestDB. **Type**: `string` ```yaml # Examples: table: trades ``` ### [](#timestamp_string_fields)`timestamp_string_fields[]` String fields with textual timestamps. **Type**: `array` ### [](#timestamp_string_format)`timestamp_string_format` The timestamp format, which is used when parsing timestamp string fields and uses Golang’s time formatting. **Type**: `string` **Default**: `Jan _2 15:04:05.000000Z0700` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#token)`token` The bearer token to use for authentication, which takes precedence over the basic authentication username and password. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#username)`username` The username to use for basic authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` --- # Page 177: redis_hash **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/redis_hash.md --- # redis\_hash --- title: redis_hash latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/redis_hash page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/redis_hash.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/redis_hash.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/redis_hash/ "View the Cloud version of this component"), Self-Managed Sets Redis hash objects using the HMSET command. #### Common ```yml outputs: label: "" redis_hash: url: "" # No default (required) key: "" # No default (required) walk_metadata: false walk_json_object: false fields: {} max_in_flight: 64 ``` #### Advanced ```yml outputs: label: "" redis_hash: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] key: "" # No default (required) walk_metadata: false walk_json_object: false fields: {} max_in_flight: 64 ``` The field `key` supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries), allowing you to create a unique key for each message. The field `fields` allows you to specify an explicit map of field names to interpolated values, also evaluated per message of a batch: ```yaml output: redis_hash: url: tcp://localhost:6379 key: ${!json("id")} fields: topic: ${!meta("kafka_topic")} partition: ${!meta("kafka_partition")} content: ${!json("document.text")} ``` If the field `walk_metadata` is set to `true` then Redpanda Connect will walk all metadata fields of messages and add them to the list of hash fields to set. If the field `walk_json_object` is set to `true` then Redpanda Connect will walk each message as a JSON object, extracting keys and the string representation of their value and adds them to the list of hash fields to set. The order of hash field extraction is as follows: 1. Metadata (if enabled) 2. JSON object (if enabled) 3. Explicit fields Where latter stages will overwrite matching field names of a former stage. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ## [](#fields)Fields ### [](#client_name)`client_name` Set the client name for the Redis connection. Requires version 4.82.0 or later. **Type**: `string` **Default**: `redpanda-connect` ### [](#fields-2)`fields` A map of key/value pairs to set as hash fields. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `{}` ### [](#key)`key` The key for each message, function interpolations should be used to create a unique key per message. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: key: ${! @.kafka_key } # --- key: ${! this.doc.id } # --- key: ${! counter() } ``` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` ### [](#walk_json_object)`walk_json_object` Whether to walk each message as a JSON object and add each key/value pair to the list of hash fields to set. **Type**: `bool` **Default**: `false` ### [](#walk_metadata)`walk_metadata` Whether all metadata fields of messages should be walked and added to the list of hash fields to set. **Type**: `bool` **Default**: `false` --- # Page 178: redis_list **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/redis_list.md --- # redis\_list --- title: redis_list latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/redis_list page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/redis_list.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/redis_list.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/redis_list/)[Input](/redpanda-connect/components/inputs/redis_list/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/redis_list/ "View the Cloud version of this component"), Self-Managed Pushes messages onto the end of a Redis list (which is created if it doesn’t already exist) using the RPUSH command. #### Common ```yml outputs: label: "" redis_list: url: "" # No default (required) key: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" redis_list: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] key: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) command: rpush ``` The field `key` supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries), allowing you to create a unique key for each message. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](../../../configuration/batching/). ## [](#fields)Fields ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#client_name)`client_name` Set the client name for the Redis connection. Requires version 4.82.0 or later. **Type**: `string` **Default**: `redpanda-connect` ### [](#command)`command` The command used to push elements to the Redis list Requires version 4.22.0 or later. **Type**: `string` **Default**: `rpush` **Options**: `rpush`, `lpush` ### [](#key)`key` The key for each message, function interpolations can be optionally used to create a unique key per message. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: key: some_list # --- key: ${! @.kafka_key } # --- key: ${! this.doc.id } # --- key: ${! counter() } ``` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 179: redis_pubsub **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/redis_pubsub.md --- # redis\_pubsub --- title: redis_pubsub latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/redis_pubsub page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/redis_pubsub.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/redis_pubsub.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/redis_pubsub/)[Input](/redpanda-connect/components/inputs/redis_pubsub/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/redis_pubsub/ "View the Cloud version of this component"), Self-Managed Publishes messages through the Redis PubSub model. It is not possible to guarantee that messages have been received. #### Common ```yml outputs: label: "" redis_pubsub: url: "" # No default (required) channel: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" redis_pubsub: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] channel: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` This output will interpolate functions within the channel field, you can find a list of functions [here](../../../configuration/interpolation/#bloblang-queries). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](../../../configuration/batching/). ## [](#fields)Fields ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#channel)`channel` The channel to publish messages to. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#client_name)`client_name` Set the client name for the Redis connection. Requires version 4.82.0 or later. **Type**: `string` **Default**: `redpanda-connect` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 180: redis_streams **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/redis_streams.md --- # redis\_streams --- title: redis_streams latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/redis_streams page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/redis_streams.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/redis_streams.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/redis_streams/)[Input](/redpanda-connect/components/inputs/redis_streams/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/redis_streams/ "View the Cloud version of this component"), Self-Managed Pushes messages to a Redis (v5.0+) Stream (which is created if it doesn’t already exist) using the XADD command. #### Common ```yml outputs: label: "" redis_streams: url: "" # No default (required) stream: "" # No default (required) id: * body_key: body max_length: 0 max_in_flight: 64 metadata: exclude_prefixes: [] batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" redis_streams: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] stream: "" # No default (required) id: * body_key: body max_length: 0 max_in_flight: 64 metadata: exclude_prefixes: [] batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` It’s possible to specify a maximum length of the target stream by setting it to a value greater than 0, in which case this cap is applied only when Redis is able to remove a whole macro node, for efficiency. Redis stream entries are key/value pairs, as such it is necessary to specify the key to be set to the body of the message. All metadata fields of the message will also be set as key/value pairs, if there is a key collision between a metadata item and the body then the body takes precedence. ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](../../../configuration/batching/). ## [](#fields)Fields ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#body_key)`body_key` A key to set the raw body of the message to. **Type**: `string` **Default**: `body` ### [](#client_name)`client_name` Set the client name for the Redis connection. Requires version 4.82.0 or later. **Type**: `string` **Default**: `redpanda-connect` ### [](#id)`id` The entry ID for the stream message. Allows function interpolations. When set to `*` (the default), Redis auto-generates a unique ID based on the current time. Set a custom ID to control message ordering, for example to replay messages in upstream order. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `*` ```yaml # Examples: id: * # --- id: ${! @redis_stream } # --- id: ${! this.id } # --- id: ${! counter() }-0 ``` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#max_length)`max_length` When greater than zero enforces a rough cap on the length of the target stream. **Type**: `int` **Default**: `0` ### [](#metadata)`metadata` Specify criteria for which metadata values are included in the message body. **Type**: `object` ### [](#metadata-exclude_prefixes)`metadata.exclude_prefixes[]` Provide a list of explicit metadata key prefixes to be excluded when adding metadata to sent messages. **Type**: `array` **Default**: `[]` ### [](#stream)`stream` The stream to add messages to. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 181: redpanda_common **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/redpanda_common.md --- # redpanda\_common --- title: redpanda_common latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/redpanda_common page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/redpanda_common.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/redpanda_common.adoc page-git-created-date: "2024-11-19" page-git-modified-date: "2025-11-03" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/redpanda_common/)[Input](/redpanda-connect/components/inputs/redpanda_common/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/redpanda_common/ "View the Cloud version of this component"), Self-Managed **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. > ⚠️ **WARNING: Deprecated in 4.68.0** > > Deprecated in 4.68.0 > > This component is deprecated and will be removed in the next major version release. Please consider moving onto the unified [`redpanda` input](../../inputs/redpanda/) and [`redpanda` output](../redpanda/) components. Sends data to a Redpanda (Kafka) broker, using credentials from a common `redpanda` configuration block. To avoid duplicating Redpanda cluster credentials in your `redpanda_common` input, output, or any other components in your data pipeline, you can use a single [`redpanda` configuration block](../../redpanda/about/). For more details, see the [Pipeline example](#pipeline-example). Introduced in version 4.39.0. > 📝 **NOTE** > > If you need to move topic data between Redpanda clusters or other Apache Kafka clusters, consider using the [`redpanda` input](../../inputs/redpanda/) and [output](../redpanda/) instead. #### Common ```yml outputs: label: "" redpanda_common: topic: "" # No default (required) key: "" # No default (optional) partition: "" # No default (optional) metadata: include_prefixes: [] include_patterns: [] max_in_flight: 10 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" redpanda_common: topic: "" # No default (required) key: "" # No default (optional) partition: "" # No default (optional) metadata: include_prefixes: [] include_patterns: [] timestamp_ms: "" # No default (optional) max_in_flight: 10 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#pipeline-example)Pipeline example This data pipeline reads data from `topic_A` and `topic_B` on a Redpanda cluster, and then writes the data to `topic_C` on the same cluster. The cluster details are configured within the `redpanda` configuration block, so you only need to configure them once. This is a useful feature when you have multiple inputs and outputs in the same data pipeline that need to connect to the same cluster. ```none input: redpanda_common: topics: [ topic_A, topic_B ] output: redpanda_common: topic: topic_C key: ${! @id } redpanda: seed_brokers: [ "127.0.0.1:9092" ] tls: enabled: true sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ## [](#fields)Fields ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#key)`key` A key to populate for each message (optional). This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this number to improve throughput until performance plateaus. **Type**: `int` **Default**: `10` ### [](#metadata)`metadata` Configure which metadata values are added to messages as headers. This allows you to pass additional context information along with your messages. **Type**: `object` ### [](#metadata-include_patterns)`metadata.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#metadata-include_prefixes)`metadata.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#partition)`partition` Set a partition for each message (optional). This field is only relevant when the `partitioner` is set to `manual`. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). You must provide an interpolation string that is a valid integer. **Type**: `string` ```yaml # Examples: partition: ${! meta("partition") } ``` ### [](#timestamp_ms)`timestamp_ms` Set a timestamp (in milliseconds) for each message (optional). When left empty, the current timestamp is used. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: timestamp_ms: ${! timestamp_unix_milli() } # --- timestamp_ms: ${! metadata("kafka_timestamp_ms") } ``` ### [](#topic)`topic` A topic to write messages to. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 182: redpanda_migrator **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/redpanda_migrator.md --- # redpanda\_migrator --- title: redpanda_migrator latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/redpanda_migrator page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/redpanda_migrator.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/redpanda_migrator.adoc page-git-created-date: "2024-10-02" page-git-modified-date: "2025-12-03" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/redpanda_migrator/)[Input](/redpanda-connect/components/inputs/redpanda_migrator/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/redpanda_migrator/ "View the Cloud version of this component"), Self-Managed A Kafka producer for migrating data between Kafka/Redpanda clusters. The `redpanda_migrator` output coordinates migration of topics, schemas, and consumer groups from a source Kafka/Redpanda cluster to a destination cluster. > ❗ **IMPORTANT** > > This output **must** be paired with a [`redpanda_migrator` input](../../inputs/redpanda_migrator/) in the same pipeline. Each pipeline requires both input and output components. Introduced in version 4.37.0. #### Common ```yml outputs: label: "" redpanda_migrator: seed_brokers: [] # No default (required) schema_registry: url: "" # No default (required) timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} enabled: true interval: 5m include: [] # No default (optional) exclude: [] # No default (optional) subject: "" # No default (optional) versions: all include_deleted: false translate_ids: false normalize: false strict: false max_parallel_http_requests: 10 consumer_groups: enabled: true interval: 1m fetch_timeout: 10s include: [] # No default (optional) exclude: [] # No default (optional) only_empty: false topic: ${! @kafka_topic } topic_replication_factor: "" # No default (optional) sync_topic_acls: false max_in_flight: 10 ``` #### Advanced ```yml outputs: label: "" redpanda_migrator: seed_brokers: [] # No default (required) client_id: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) metadata_max_age: 1m request_timeout_overhead: 10s conn_idle_timeout: 20s tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s partitioner: "" # No default (optional) idempotent_write: true compression: "" # No default (optional) allow_auto_topic_creation: true timeout: 10s max_message_bytes: 1MiB broker_write_max_bytes: 100MiB schema_registry: url: "" # No default (required) timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} enabled: true interval: 5m include: [] # No default (optional) exclude: [] # No default (optional) subject: "" # No default (optional) versions: all include_deleted: false translate_ids: false normalize: false strict: false max_parallel_http_requests: 10 consumer_groups: enabled: true interval: 1m fetch_timeout: 10s include: [] # No default (optional) exclude: [] # No default (optional) only_empty: false topic: ${! @kafka_topic } topic_replication_factor: "" # No default (optional) sync_topic_interval: 5m sync_topic_acls: false serverless: false provenance_header: redpanda-migrator-provenance offset_header: redpanda-migrator-offset max_in_flight: 10 ``` ## [](#multiple-migrator-pairs)Multiple migrator pairs When using multiple migrator pairs in a pipeline, match the `label` field exactly between input and output components for correct coordination. ## [](#performance-tuning)Performance tuning For high-throughput workloads, adjust the following settings: On this output component: - `max_in_flight`: Set to the total number of partitions being copied in parallel (up to all partitions in the cluster) On the paired [`redpanda_migrator` input component](../../inputs/redpanda_migrator/#performance-tuning): - `partition_buffer_bytes`: Set to 2MB to increase per-partition buffer size - `max_yield_batch_bytes`: Set to 1MB to allow larger batches to be yielded ## [](#synchronization-details)Synchronization details **Topics** - Name resolution with interpolation (default: preserve source name) - Automatic creation with mirrored partition counts - Selectable replication factor (default: inherit from source) - Supported topic configuration keys (serverless-aware subset) - Optional ACL replication: - Excludes `ALLOW WRITE` entries - Downgrades `ALLOW ALL` to `READ` - Preserves resource pattern type and host filters **Schema Registry** - One-shot or periodic syncing - Subject selection via include/exclude regex - Subject renaming with interpolation - Versions: `latest` or `all` (default: `all`) - Optional include of soft-deleted subjects - ID handling: translate IDs or keep fixed - Optional schema normalization - Compatibility propagation (per-subject only) - Schema metadata/rules not copied in Serverless mode **Consumer Groups** - Periodic syncing - Group selection using regex - Only `Empty` state groups migrated - Timestamp-based offset translation (approximate) - No rewind guarantee: offsets only move forward - Requires matching partition counts ## [](#how-it-works)How it works - Topics: Synced on demand. First write triggers creation. - Schema Registry: Synced at connect, then as needed. - Consumer Groups: Background loop, filtered by topic mappings. ## [](#guarantees)Guarantees - Topics created with intended partitioning/replication. - Existing topics respected. Mismatches logged. - Consumer group offsets never rewound. - ACL replication excludes unsafe grants. ## [](#limitations)Limitations - Destination Schema Registry must be in `READWRITE` or `IMPORT` mode. - Offset translation is best-effort. - Consumer group migration requires identical partition counts. ## [](#metrics)Metrics The component exposes comprehensive metrics for monitoring migration operations: | Metric Name | Type | Labels | Description | | --- | --- | --- | --- | | Topic migration metrics | | | | | redpanda_migrator_topics_created_total | counter | | Total topics created on destination | | redpanda_migrator_topic_create_errors_total | counter | | Topic creation errors | | redpanda_migrator_topic_create_latency_ns | timer | | Topic creation latency (ns) | | Schema Registry migration metrics | | | | | redpanda_migrator_sr_schemas_created_total | counter | | Schemas created in destination registry | | redpanda_migrator_sr_schema_create_errors_total | counter | | Schema creation errors | | redpanda_migrator_sr_schema_create_latency_ns | timer | | Schema creation latency (ns) | | redpanda_migrator_sr_compatibility_updates_total | counter | | Compatibility level updates applied | | redpanda_migrator_sr_compatibility_update_errors_total | counter | | Compatibility update errors | | redpanda_migrator_sr_compatibility_update_latency_ns | timer | | Compatibility update latency (ns) | | Consumer group migration metrics | | | | | redpanda_migrator_cg_offsets_translated_total | counter | group | Offsets translated per consumer group | | redpanda_migrator_cg_offset_translation_errors_total | counter | group | Offset translation errors per group | | redpanda_migrator_cg_offset_translation_latency_ns | timer | group | Offset translation latency per group (ns) | | redpanda_migrator_cg_offsets_committed_total | counter | group | Offsets committed per consumer group | | redpanda_migrator_cg_offset_commit_errors_total | counter | group | Offset commit errors per group | | redpanda_migrator_cg_offset_commit_latency_ns | timer | group | Offset commit latency per group (ns) | | Consumer lag metrics | | | | | redpanda_lag | gauge | topic, partition | Current consumer lag in messages for each topic partition. Shows difference between high water mark and current consumer position. | ## [](#examples)Examples ### [](#basic-migration)Basic migration Migrate topics, schemas and consumer groups from source to destination. ```yaml input: redpanda_migrator: seed_brokers: ["source:9092"] topics: ["orders", "payments"] consumer_group: "migration" output: redpanda_migrator: seed_brokers: ["destination:9092"] # Write to the same topic name topic: ${! metadata("kafka_topic") } schema_registry: url: "http://dest-registry:8081" translate_ids: true consumer_groups: interval: 1m ``` ### [](#migration-to-redpanda-serverless)Migration to Redpanda Serverless Migrate from Confluent/Kafka to Redpanda Cloud serverless cluster with authentication. ```yaml input: redpanda_migrator: seed_brokers: ["source-kafka:9092"] regexp_topics_include: - '.' regexp_topics_exclude: - '^_' consumer_group: "migrator_cg" schema_registry: url: "http://source-registry:8081" output: redpanda_migrator: seed_brokers: ["serverless-cluster.redpanda.com:9092"] tls: enabled: true sasl: - mechanism: SCRAM-SHA-256 username: "migrator" password: "migrator" schema_registry: url: "https://serverless-cluster.redpanda.com:8081" basic_auth: enabled: true username: "migrator" password: "migrator" translate_ids: true consumer_groups: exclude: - "migrator_cg" # Exclude the migration consumer group itself serverless: true # Enable serverless mode for restricted configurations ``` ## [](#fields)Fields ### [](#allow_auto_topic_creation)`allow_auto_topic_creation` Enables topics to be auto created if they do not exist when fetching their metadata. **Type**: `bool` **Default**: `true` ### [](#broker_write_max_bytes)`broker_write_max_bytes` The maximum number of bytes this output can write to a broker connection in a single write. This field corresponds to Kafka’s `socket.request.max.bytes`. **Type**: `string` **Default**: `100MiB` ```yaml # Examples: broker_write_max_bytes: 128MB # --- broker_write_max_bytes: 50mib ``` ### [](#client_id)`client_id` An identifier for the client connection. **Type**: `string` **Default**: `redpanda-connect` ### [](#compression)`compression` Set an explicit compression type (optional). The default preference is to use `snappy` when the broker supports it. Otherwise, use `none`. **Type**: `string` **Options**: `lz4`, `snappy`, `gzip`, `none`, `zstd` ### [](#conn_idle_timeout)`conn_idle_timeout` The maximum duration that connections can remain idle before they are automatically closed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `20s` ### [](#consumer_groups)`consumer_groups` **Type**: `object` ### [](#consumer_groups-enabled)`consumer_groups.enabled` Whether consumer group offset migration is enabled. When disabled, no consumer group operations are performed. **Type**: `bool` **Default**: `true` ### [](#consumer_groups-exclude)`consumer_groups.exclude[]` Regular expressions for consumer groups to exclude from offset migration. Takes precedence over include patterns. Useful for excluding system or temporary groups. **Type**: `array` ```yaml # Examples: exclude: [".*-test", ".*-temp", "connect-.*"] # --- exclude: ["dev-.*", "local-.*"] ``` ### [](#consumer_groups-fetch_timeout)`consumer_groups.fetch_timeout` Maximum time to wait for data when fetching records for timestamp-based offset translation. Increase for clusters with low message throughput. **Type**: `string` **Default**: `10s` ```yaml # Examples: fetch_timeout: 1s # Fast clusters # --- fetch_timeout: 10s # Slower clusters ``` ### [](#consumer_groups-include)`consumer_groups.include[]` Regular expressions for consumer groups to include in offset migration. If empty, all groups are included (unless excluded). **Type**: `array` ```yaml # Examples: include: ["prod-.*", "staging-.*"] # --- include: ["app-.*", "service-.*"] ``` ### [](#consumer_groups-interval)`consumer_groups.interval` How often to synchronise consumer group offsets. Regular syncing helps maintain offset accuracy during ongoing migration. **Type**: `string` **Default**: `1m` ```yaml # Examples: interval: 0s # Disabled # --- interval: 30s # Sync every 30 seconds # --- interval: 5m # Sync every 5 minutes ``` ### [](#consumer_groups-only_empty)`consumer_groups.only_empty` Whether to only migrate Empty consumer groups. When false (default), all statuses except Dead are included; when true, only Empty groups are migrated. **Type**: `bool` **Default**: `false` ### [](#idempotent_write)`idempotent_write` Enable the idempotent write producer option. This requires the `IDEMPOTENT_WRITE` permission on `CLUSTER`. Disable this option if the `IDEMPOTENT_WRITE` permission is unavailable. **Type**: `bool` **Default**: `true` ### [](#max_in_flight)`max_in_flight` The maximum number of batches to send in parallel at any given time. Increase this value to improve throughput during migration. For optimal performance, set this to match the total number of partitions being migrated. Setting it higher than the partition count provides no additional benefit, as each partition can only have one in-flight batch at a time. Example: If migrating 100 partitions, set `max_in_flight: 100` for maximum throughput. **Type**: `int` **Default**: `10` ```yaml # Examples: max_in_flight: 64 # For a cluster with 64 partitions # --- max_in_flight: 128 # For multiple topics with combined 128 partitions ``` ### [](#max_message_bytes)`max_message_bytes` The maximum space in bytes that an individual message may use. Messages larger than this value are rejected. This field corresponds to Kafka’s `max.message.bytes`. **Type**: `string` **Default**: `1MiB` ```yaml # Examples: max_message_bytes: 100MB # --- max_message_bytes: 50mib ``` ### [](#metadata_max_age)`metadata_max_age` The maximum period of time after which metadata is refreshed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Lower values provide more responsive topic and partition discovery but may increase broker load. Higher values reduce broker queries but can delay detection of topology changes. **Type**: `string` **Default**: `1m` ### [](#offset_header)`offset_header` The name of a message header to add to migrated records. This header contains the source offset, enabling exact consumer group offset translation during migration. When left empty (default), no offset header is added and consumer groups are migrated using timestamp-based positioning. This approach works well for most cases, but may be imprecise for consumer groups with no committed offsets when multiple records share the same timestamp (timestamps have millisecond resolution). Set this field to enable precise offset translation, especially when migrating consumer groups that are caught up or have minimal lag. Note: This header is only added when consumer group migration is enabled. **Type**: `string` **Default**: `redpanda-migrator-offset` ### [](#partitioner)`partitioner` Override the default murmur2 hashing partitioner. **Type**: `string` | Option | Summary | | --- | --- | | least_backup | Chooses the least backed up partition (the partition with the fewest amount of buffered records). Partitions are selected per batch. | | manual | Manually select a partition for each message, requires the field partition to be specified. | | murmur2_hash | Kafka’s default hash algorithm that uses a 32-bit murmur2 hash of the key to compute which partition the record will be on. | | round_robin | Round-robin’s messages through all available partitions. This algorithm has lower throughput and causes higher CPU load on brokers, but can be useful if you want to ensure an even distribution of records to partitions. | ### [](#provenance_header)`provenance_header` Header name to add to migrated records indicating their source cluster. When set, each migrated message receives a header with this name containing the source cluster’s seed broker addresses, enabling downstream systems to track message origins for auditing, debugging, or multi-cluster orchestration workflows. If empty, no provenance header is added to messages. The header value format is a comma-separated list of the source cluster’s `seed_brokers`. Example: Setting `provenance_header: "rp-source-cluster"` adds a header like `rp-source-cluster: "kafka-1:9092,kafka-2:9092"`. **Type**: `string` **Default**: `redpanda-migrator-provenance` ### [](#request_timeout_overhead)`request_timeout_overhead` Grants an additional buffer or overhead to requests that have timeout fields defined. This field is based on the behavior of Apache Kafka’s `request.timeout.ms` parameter, but with the option to extend the timeout deadline. **Type**: `string` **Default**: `10s` ### [](#sasl)`sasl[]` Specify one or more methods of SASL authentication, which are tried in order. If the broker supports the first mechanism, all connections will use that mechanism. If the first mechanism fails, the client picks the first supported mechanism. Connections fail if the broker does not support any client mechanisms. **Type**: `object` ```yaml # Examples: sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ### [](#sasl-aws)`sasl[].aws` Contains AWS specific fields for when the `mechanism` is set to `AWS_MSK_IAM`. **Type**: `object` ### [](#sasl-aws-credentials)`sasl[].aws.credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](../../../guides/cloud/aws/). **Type**: `object` ### [](#sasl-aws-credentials-from_ec2_role)`sasl[].aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#sasl-aws-credentials-id)`sasl[].aws.credentials.id` The ID of credentials to use. **Type**: `string` ### [](#sasl-aws-credentials-profile)`sasl[].aws.credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#sasl-aws-credentials-role)`sasl[].aws.credentials.role` A role ARN to assume. **Type**: `string` ### [](#sasl-aws-credentials-role_external_id)`sasl[].aws.credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#sasl-aws-credentials-secret)`sasl[].aws.credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#sasl-aws-credentials-token)`sasl[].aws.credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#sasl-aws-endpoint)`sasl[].aws.endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#sasl-aws-region)`sasl[].aws.region` The AWS region to target. **Type**: `string` ### [](#sasl-aws-tcp)`sasl[].aws.tcp` TCP socket configuration. **Type**: `object` ### [](#sasl-aws-tcp-connect_timeout)`sasl[].aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-aws-tcp-keep_alive)`sasl[].aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#sasl-aws-tcp-keep_alive-count)`sasl[].aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#sasl-aws-tcp-keep_alive-idle)`sasl[].aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-keep_alive-interval)`sasl[].aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-tcp_user_timeout)`sasl[].aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-extensions)`sasl[].extensions` Key/value pairs to add to OAUTHBEARER authentication requests. **Type**: `string` ### [](#sasl-mechanism)`sasl[].mechanism` The SASL mechanism to use. **Type**: `string` | Option | Summary | | --- | --- | | AWS_MSK_IAM | AWS IAM based authentication as specified by the 'aws-msk-iam-auth' java library. | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. | | REDPANDA_CLOUD_SERVICE_ACCOUNT | Redpanda Cloud Service Account authentication when running in Redpanda Cloud. | | SCRAM-SHA-256 | SCRAM based authentication as specified in RFC5802. | | SCRAM-SHA-512 | SCRAM based authentication as specified in RFC5802. | | none | Disable sasl authentication | ### [](#sasl-password)`sasl[].password` A password to provide for PLAIN or SCRAM-\* authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#sasl-token)`sasl[].token` The token to use for a single session’s OAUTHBEARER authentication. **Type**: `string` **Default**: `""` ### [](#sasl-username)`sasl[].username` A username to provide for PLAIN or SCRAM-\* authentication. **Type**: `string` **Default**: `""` ### [](#schema_registry)`schema_registry` Configuration for schema registry integration. Enables migration of schema subjects, versions, and compatibility settings between clusters. **Type**: `object` ### [](#schema_registry-basic_auth)`schema_registry.basic_auth` Allows you to specify basic authentication. **Type**: `object` ### [](#schema_registry-basic_auth-enabled)`schema_registry.basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-basic_auth-password)`schema_registry.basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-basic_auth-username)`schema_registry.basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#schema_registry-enabled)`schema_registry.enabled` Whether schema registry migration is enabled. When disabled, no schema operations are performed. **Type**: `bool` **Default**: `true` ### [](#schema_registry-exclude)`schema_registry.exclude[]` Regular expressions for schema subjects to exclude from migration. Takes precedence over include patterns. Note: the migrator consumer group is always ignored. **Type**: `array` ```yaml # Examples: exclude: [".*-test", ".*-temp"] # --- exclude: ["dev-.*", "local-.*"] ``` ### [](#schema_registry-include)`schema_registry.include[]` Regular expressions for schema subjects to include in migration. If empty, all subjects are included (unless excluded). Note: the migrator consumer group is always ignored. **Type**: `array` ```yaml # Examples: include: ["prod-.*", "staging-.*"] # --- include: ["user-.*", "order-.*"] ``` ### [](#schema_registry-include_deleted)`schema_registry.include_deleted` Whether to include soft-deleted schemas in migration. Useful for complete migration but may not be supported by all schema registries. **Type**: `bool` **Default**: `false` ### [](#schema_registry-interval)`schema_registry.interval` How often to synchronise schema registry subjects. Set to 0s for one-time sync at startup only. **Type**: `string` **Default**: `5m` ```yaml # Examples: interval: 0s # One-time sync only # --- interval: 5m # Sync every 5 minutes # --- interval: 30m # Sync every 30 minutes ``` ### [](#schema_registry-jwt)`schema_registry.jwt` Beta Allows you to specify JWT authentication. **Type**: `object` ### [](#schema_registry-jwt-claims)`schema_registry.jwt.claims` A value used to identify the claims that issued the JWT. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-enabled)`schema_registry.jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-jwt-headers)`schema_registry.jwt.headers` Add optional key/value headers to the JWT. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-private_key_file)`schema_registry.jwt.private_key_file` A file with the PEM encoded via PKCS1 or PKCS8 as private key. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt-signing_method)`schema_registry.jwt.signing_method` A method used to sign the token such as RS256, RS384, RS512 or EdDSA. **Type**: `string` **Default**: `""` ### [](#schema_registry-max_parallel_http_requests)`schema_registry.max_parallel_http_requests` Maximum number of parallel HTTP requests to the schema registry. Controls concurrency when syncing multiple schemas. **Type**: `int` **Default**: `10` ### [](#schema_registry-normalize)`schema_registry.normalize` Whether to normalize schemas when creating them in the destination registry. **Type**: `bool` **Default**: `false` ### [](#schema_registry-oauth)`schema_registry.oauth` Allows you to specify open authentication via OAuth version 1. **Type**: `object` ### [](#schema_registry-oauth-access_token)`schema_registry.oauth.access_token` A value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-access_token_secret)`schema_registry.oauth.access_token_secret` A secret provided in order to establish ownership of a given access token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_key)`schema_registry.oauth.consumer_key` A value used to identify the client to the service provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_secret)`schema_registry.oauth.consumer_secret` A secret used to establish ownership of the consumer key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-enabled)`schema_registry.oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-strict)`schema_registry.strict` Error on unknown schema IDs. Only relevant when translate\_ids is true. When false (default), unknown schema IDs are passed through unchanged, allowing migration of topics with mixed message formats. Note: messages with 0-byte prefixes (e.g., protobuf) cannot be distinguished from schema registry headers and may fail when strict is enabled. **Type**: `bool` **Default**: `false` ### [](#schema_registry-subject)`schema_registry.subject` Template for transforming subject names during migration. Use interpolation to rename subjects systematically. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: subject: prod_${! metadata("schema_registry_subject") } # --- subject: ${! metadata("schema_registry_subject") | replace("dev_", "prod_") } ``` ### [](#schema_registry-timeout)`schema_registry.timeout` HTTP client timeout for schema registry requests. **Type**: `string` **Default**: `5s` ### [](#schema_registry-tls)`schema_registry.tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#schema_registry-tls-client_certs)`schema_registry.tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#schema_registry-tls-client_certs-cert)`schema_registry.tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-cert_file)`schema_registry.tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key)`schema_registry.tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key_file)`schema_registry.tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-password)`schema_registry.tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#schema_registry-tls-enable_renegotiation)`schema_registry.tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-enabled)`schema_registry.tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-root_cas)`schema_registry.tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#schema_registry-tls-root_cas_file)`schema_registry.tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#schema_registry-tls-skip_cert_verify)`schema_registry.tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#schema_registry-translate_ids)`schema_registry.translate_ids` Whether to translate schema IDs during migration. **Type**: `bool` **Default**: `false` ### [](#schema_registry-url)`schema_registry.url` The base URL of the schema registry service. Required for schema migration functionality. **Type**: `string` ```yaml # Examples: url: http://localhost:8081 # --- url: https://schema-registry.example.com:8081 ``` ### [](#schema_registry-versions)`schema_registry.versions` Which schema versions to migrate. 'latest' migrates only the current version, 'all' migrates complete version history for better compatibility. **Type**: `string` **Default**: `all` **Options**: `latest`, `all` ### [](#seed_brokers)`seed_brokers[]` A list of broker addresses to connect to. Use commas to separate multiple addresses in a single list item. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" # --- seed_brokers: - "foo:9092" - "bar:9092" # --- seed_brokers: - "foo:9092,bar:9092" ``` ### [](#serverless)`serverless` Enable serverless mode for Redpanda Cloud serverless clusters. This restricts topic configurations and schema features to those supported by serverless environments. **Type**: `bool` **Default**: `false` ### [](#sync_topic_acls)`sync_topic_acls` Whether to synchronise topic ACLs from source to destination cluster. ACLs are transformed safely: ALLOW WRITE permissions are excluded, and ALLOW ALL is downgraded to ALLOW READ to prevent conflicts. **Type**: `bool` **Default**: `false` ### [](#sync_topic_interval)`sync_topic_interval` How often to synchronize topics from the source cluster to the destination. This creates destination topics for any new source topics, including empty topics with no message flow. Set to 0s to disable periodic sync (topics are still created on first message). **Type**: `string` **Default**: `5m` ```yaml # Examples: sync_topic_interval: 0s # Disable periodic sync # --- sync_topic_interval: 1m # Sync every minute # --- sync_topic_interval: 5m # Sync every 5 minutes ``` ### [](#tcp)`tcp` TCP socket configuration. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` The maximum period of time to wait for message sends before abandoning the request and retrying. **Type**: `string` **Default**: `10s` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic)`topic` A topic to write messages to. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `${! @kafka_topic }` ```yaml # Examples: topic: prod_${! @kafka_topic } ``` ### [](#topic_replication_factor)`topic_replication_factor` The replication factor for created topics. If not specified, inherits the replication factor from source topics. Useful when migrating to clusters with different sizes. **Type**: `int` ```yaml # Examples: topic_replication_factor: 3 # --- topic_replication_factor: 1 # For single-node clusters ``` --- # Page 183: redpanda **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/redpanda.md --- # redpanda --- title: redpanda latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/redpanda page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/redpanda.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/redpanda.adoc page-git-created-date: "2024-11-19" page-git-modified-date: "2025-10-06" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/redpanda/)[Cache](/redpanda-connect/components/caches/redpanda/)[Input](/redpanda-connect/components/inputs/redpanda/)[Tracer](/redpanda-connect/components/tracers/redpanda/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/redpanda/ "View the Cloud version of this component"), Self-Managed Sends message data to Kafka brokers and waits for acknowledgement before propagating any acknowledgements back to the input. Introduced in version 4.39.0. > 📝 **NOTE** > > If you intend to use multiple `redpanda` components in your data pipeline, which have a common set of credentials, consider using the [`redpanda_common` input](../../inputs/redpanda_common/) and [output](../redpanda_common/) instead. #### Common ```yml outputs: label: "" redpanda: seed_brokers: [] # No default (optional) topic: "" # No default (required) key: "" # No default (optional) partition: "" # No default (optional) metadata: include_prefixes: [] include_patterns: [] max_in_flight: 256 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" redpanda: seed_brokers: [] # No default (optional) client_id: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) metadata_max_age: 1m request_timeout_overhead: 10s conn_idle_timeout: 20s tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s topic: "" # No default (required) key: "" # No default (optional) partition: "" # No default (optional) metadata: include_prefixes: [] include_patterns: [] timestamp_ms: "" # No default (optional) max_in_flight: 256 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) inject_tracing_map: "" # No default (optional) partitioner: "" # No default (optional) idempotent_write: true compression: "" # No default (optional) allow_auto_topic_creation: true timeout: 10s max_message_bytes: 1MiB broker_write_max_bytes: 100MiB ``` ## [](#fields)Fields ### [](#allow_auto_topic_creation)`allow_auto_topic_creation` Enables topics to be auto created if they do not exist when fetching their metadata. **Type**: `bool` **Default**: `true` ### [](#batching)`batching` Optional explicit batching policy for the output. Note that when batches are formed at the input level they can be expanded by this policy, but not contracted. When consuming data from a Redpanda input it is recommended to tune batches from the input config via the `max_yield_batch_bytes` field, or the `unordered_processing.batching` field if appropriate. **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#broker_write_max_bytes)`broker_write_max_bytes` The maximum number of bytes this output can write to a broker connection in a single write. This field corresponds to Kafka’s `socket.request.max.bytes`. **Type**: `string` **Default**: `100MiB` ```yaml # Examples: broker_write_max_bytes: 128MB # --- broker_write_max_bytes: 50mib ``` ### [](#client_id)`client_id` An identifier for the client connection. **Type**: `string` **Default**: `redpanda-connect` ### [](#compression)`compression` Set an explicit compression type (optional). The default preference is to use `snappy` when the broker supports it. Otherwise, use `none`. **Type**: `string` **Options**: `lz4`, `snappy`, `gzip`, `none`, `zstd` ### [](#conn_idle_timeout)`conn_idle_timeout` The maximum duration that connections can remain idle before they are automatically closed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `20s` ### [](#idempotent_write)`idempotent_write` Enable the idempotent write producer option. This requires the `IDEMPOTENT_WRITE` permission on `CLUSTER`. Disable this option if the `IDEMPOTENT_WRITE` permission is not available. **Type**: `bool` **Default**: `true` ### [](#inject_tracing_map)`inject_tracing_map` EXPERIMENTAL: A [Bloblang mapping](../../../guides/bloblang/about/) used to inject an object containing tracing propagation information into outbound messages. The specification of the injected fields will match the format used by the service wide tracer. Requires version 3.45.0 or later. **Type**: `string` ```yaml # Examples: inject_tracing_map: meta = @.merge(this) # --- inject_tracing_map: root.meta.span = this ``` ### [](#key)`key` An optional key to populate for each message. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this number to improve throughput until performance plateaus. **Type**: `int` **Default**: `256` ### [](#max_message_bytes)`max_message_bytes` The maximum space (in bytes) that an individual message may use. Messages larger than this value are rejected. This field corresponds to Kafka’s `max.message.bytes`. **Type**: `string` **Default**: `1MiB` ```yaml # Examples: max_message_bytes: 100MB # --- max_message_bytes: 50mib ``` ### [](#metadata)`metadata` Configure which metadata values are added to messages as headers. This allows you to pass additional context information along with your messages. **Type**: `object` ### [](#metadata-include_patterns)`metadata.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#metadata-include_prefixes)`metadata.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#metadata_max_age)`metadata_max_age` The maximum period of time after which metadata is refreshed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. Lower values provide more responsive topic and partition discovery but may increase broker load. Higher values reduce broker queries but can delay detection of topology changes. **Type**: `string` **Default**: `1m` ### [](#partition)`partition` Set a partition for each message (optional). This field is only relevant when the `partitioner` is set to `manual`. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). You must provide an interpolation string that is a valid integer. **Type**: `string` ```yaml # Examples: partition: ${! meta("partition") } ``` ### [](#partitioner)`partitioner` Override the default murmur2 hashing partitioner. **Type**: `string` | Option | Summary | | --- | --- | | least_backup | Chooses the least backed up partition (the partition with the fewest amount of buffered records). Partitions are selected per batch. | | manual | Manually select a partition for each message, requires the field partition to be specified. | | murmur2_hash | Kafka’s default hash algorithm that uses a 32-bit murmur2 hash of the key to compute which partition the record will be on. | | round_robin | Round-robin’s messages through all available partitions. This algorithm has lower throughput and causes higher CPU load on brokers, but can be useful if you want to ensure an even distribution of records to partitions. | ### [](#request_timeout_overhead)`request_timeout_overhead` Grants an additional buffer or overhead to requests that have timeout fields defined. This field is based on the behavior of Apache Kafka’s `request.timeout.ms` parameter, but with the option to extend the timeout deadline. **Type**: `string` **Default**: `10s` ### [](#sasl)`sasl[]` Specify one or more methods or mechanisms of SASL authentication, which are attempted in order. If the broker supports the first SASL mechanism, all connections use it. If the first mechanism fails, the client picks the first supported mechanism. If the broker does not support any client mechanisms, all connections fail. **Type**: `object` ```yaml # Examples: sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ### [](#sasl-aws)`sasl[].aws` Contains AWS specific fields for when the `mechanism` is set to `AWS_MSK_IAM`. **Type**: `object` ### [](#sasl-aws-credentials)`sasl[].aws.credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](../../../guides/cloud/aws/). **Type**: `object` ### [](#sasl-aws-credentials-from_ec2_role)`sasl[].aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#sasl-aws-credentials-id)`sasl[].aws.credentials.id` The ID of credentials to use. **Type**: `string` ### [](#sasl-aws-credentials-profile)`sasl[].aws.credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#sasl-aws-credentials-role)`sasl[].aws.credentials.role` A role ARN to assume. **Type**: `string` ### [](#sasl-aws-credentials-role_external_id)`sasl[].aws.credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#sasl-aws-credentials-secret)`sasl[].aws.credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#sasl-aws-credentials-token)`sasl[].aws.credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#sasl-aws-endpoint)`sasl[].aws.endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#sasl-aws-region)`sasl[].aws.region` The AWS region to target. **Type**: `string` ### [](#sasl-aws-tcp)`sasl[].aws.tcp` TCP socket configuration. **Type**: `object` ### [](#sasl-aws-tcp-connect_timeout)`sasl[].aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-aws-tcp-keep_alive)`sasl[].aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#sasl-aws-tcp-keep_alive-count)`sasl[].aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#sasl-aws-tcp-keep_alive-idle)`sasl[].aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-keep_alive-interval)`sasl[].aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-tcp_user_timeout)`sasl[].aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-extensions)`sasl[].extensions` Key/value pairs to add to OAUTHBEARER authentication requests. **Type**: `string` ### [](#sasl-mechanism)`sasl[].mechanism` The SASL mechanism to use. **Type**: `string` | Option | Summary | | --- | --- | | AWS_MSK_IAM | AWS IAM based authentication as specified by the 'aws-msk-iam-auth' java library. | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. | | REDPANDA_CLOUD_SERVICE_ACCOUNT | Redpanda Cloud Service Account authentication when running in Redpanda Cloud. | | SCRAM-SHA-256 | SCRAM based authentication as specified in RFC5802. | | SCRAM-SHA-512 | SCRAM based authentication as specified in RFC5802. | | none | Disable sasl authentication | ### [](#sasl-password)`sasl[].password` A password to provide for PLAIN or SCRAM-\* authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#sasl-token)`sasl[].token` The token to use for a single session’s OAUTHBEARER authentication. **Type**: `string` **Default**: `""` ### [](#sasl-username)`sasl[].username` A username to provide for PLAIN or SCRAM-\* authentication. **Type**: `string` **Default**: `""` ### [](#seed_brokers)`seed_brokers[]` A list of broker addresses to connect to in order. Use commas to separate multiple addresses in a single list item. Optional when `seed_brokers` is configured in a top-level `redpanda` block. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" # --- seed_brokers: - "foo:9092" - "bar:9092" # --- seed_brokers: - "foo:9092,bar:9092" ``` ### [](#tcp)`tcp` TCP socket configuration. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` The maximum period of time to wait for message sends before abandoning the request and retrying. **Type**: `string` **Default**: `10s` ### [](#timestamp_ms)`timestamp_ms` Set a timestamp (in milliseconds) for each message (optional). When left empty, the current timestamp is used. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: timestamp_ms: ${! timestamp_unix_milli() } # --- timestamp_ms: ${! metadata("kafka_timestamp_ms") } ``` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic)`topic` A topic to write messages to. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 184: reject_errored **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/reject_errored.md --- # reject\_errored --- title: reject_errored latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/reject_errored page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/reject_errored.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/reject_errored.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/reject_errored/ "View the Cloud version of this component"), Self-Managed Rejects messages that have failed their processing steps, resulting in nack behavior at the input level, otherwise sends them to a child output. ```yml # Config fields, showing default values output: label: "" reject_errored: null # No default (required) ``` The routing of messages rejected by this output depends on the type of input it came from. For inputs that support propagating nacks upstream such as AMQP or NATS the message will be nacked. However, for inputs that are sequential such as files or Kafka the messages will simply be reprocessed from scratch. ## [](#examples)Examples ### [](#rejecting-failed-messages)Rejecting Failed Messages The most straight forward use case for this output type is to nack messages that have failed their processing steps. In this example our mapping might fail, in which case the messages that failed are rejected and will be nacked by our input: ```yaml input: nats_jetstream: urls: [ nats://127.0.0.1:4222 ] subject: foos.pending pipeline: processors: - mutation: 'root.age = this.fuzzy.age.int64()' output: reject_errored: nats_jetstream: urls: [ nats://127.0.0.1:4222 ] subject: foos.processed ``` ### [](#dlqing-failed-messages)DLQing Failed Messages Another use case for this output is to send failed messages straight into a dead-letter queue. You use it within a [fallback output](../fallback/) that allows you to specify where these failed messages should go to next. ```yaml pipeline: processors: - mutation: 'root.age = this.fuzzy.age.int64()' output: fallback: - reject_errored: http_client: url: http://foo:4195/post/might/become/unreachable retries: 3 retry_period: 1s - http_client: url: http://bar:4196/somewhere/else retries: 3 retry_period: 1s ``` --- # Page 185: reject **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/reject.md --- # reject --- title: reject latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/reject page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/reject.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/reject.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/reject/ "View the Cloud version of this component"), Self-Managed Rejects all messages, treating them as though the output destination failed to publish them. ```yml # Config fields, showing default values output: label: "" reject: "" ``` The routing of messages after this output depends on the type of input it came from. For inputs that support propagating nacks upstream such as AMQP or NATS the message will be nacked. However, for inputs that are sequential such as files or Kafka the messages will simply be reprocessed from scratch. To learn when this output could be useful, see \[the [Examples](#examples). ## [](#examples)Examples ### [](#rejecting-failed-messages)Rejecting Failed Messages This input is particularly useful for routing messages that have failed during processing, where instead of routing them to some sort of dead letter queue we wish to push the error upstream. We can do this with a switch broker: ```yaml output: switch: retry_until_success: false cases: - check: '!errored()' output: amqp_1: urls: [ amqps://guest:guest@localhost:5672/ ] target_address: queue:/the_foos - output: reject: "processing failed due to: ${! error() }" ``` --- # Page 186: resource **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/resource.md --- # resource --- title: resource latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/resource page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/resource.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/resource.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-09" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/resource/)[Input](/redpanda-connect/components/inputs/resource/)[Processor](/redpanda-connect/components/processors/resource/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/resource/ "View the Cloud version of this component"), Self-Managed Resource is an output type that channels messages to a resource output, identified by its name. ```yml # Config fields, showing default values output: resource: "" ``` Resources allow you to tidy up deeply nested configs. For example, the config: ```yaml output: broker: pattern: fan_out outputs: - kafka: addresses: [ TODO ] topic: foo - gcp_pubsub: project: bar topic: baz ``` Could also be expressed as: ```yaml output: broker: pattern: fan_out outputs: - resource: foo - resource: bar output_resources: - label: foo kafka: addresses: [ TODO ] topic: foo - label: bar gcp_pubsub: project: bar topic: baz ``` You can find out more about resources in [Resources](../../../configuration/resources/) --- # Page 187: retry **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/retry.md --- # retry --- title: retry latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/retry page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/retry.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/retry.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/retry/)[Processor](/redpanda-connect/components/processors/retry/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/retry/ "View the Cloud version of this component"), Self-Managed Attempts to write messages to a child output and if the write fails for any reason the message is retried either until success or, if the retries or max elapsed time fields are non-zero, either is reached. #### Common ```yml outputs: label: "" retry: output: "" # No default (required) ``` #### Advanced ```yml outputs: label: "" retry: max_retries: 0 backoff: initial_interval: 500ms max_interval: 3s max_elapsed_time: 0s output: "" # No default (required) ``` All messages in Redpanda Connect are always retried on an output error, but this would usually involve propagating the error back to the source of the message, whereby it would be reprocessed before reaching the output layer once again. This output type is useful whenever we wish to avoid reprocessing a message on the event of a failed send. We might, for example, have a deduplication processor that we want to avoid reapplying to the same message more than once in the pipeline. Rather than retrying the same output you may wish to retry the send using a different output target (a dead letter queue). In which case you should instead use the [`fallback`](../fallback/) output type. ## [](#fields)Fields ### [](#backoff)`backoff` Control time intervals between retry attempts. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` The initial period to wait between retry attempts. The retry interval increases for each failed attempt, up to the `backoff.max_interval` value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `500ms` ### [](#backoff-max_elapsed_time)`backoff.max_elapsed_time` The maximum period to wait before retry attempts are abandoned. If zero then no limit is used. **Type**: `string` **Default**: `0s` ### [](#backoff-max_interval)`backoff.max_interval` The maximum period to wait between retry attempts. **Type**: `string` **Default**: `3s` ### [](#max_retries)`max_retries` The maximum number of retries before giving up on the request. If set to zero there is no discrete limit. **Type**: `int` **Default**: `0` ### [](#output)`output` A child output. **Type**: `output` --- # Page 188: salesforce_sink **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/salesforce_sink.md --- # salesforce\_sink --- title: salesforce_sink latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/salesforce_sink page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/salesforce_sink.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/salesforce_sink.adoc description: Writes messages to Salesforce, routing each Kafka topic to its own sObject configuration. page-git-created-date: "2026-03-31" page-git-modified-date: "2026-03-31" --- **Available in:** Self-Managed **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Writes messages to Salesforce, routing each Kafka topic to its own sObject configuration. Introduced in version 4.85.0. Consumes batches of messages and writes them to Salesforce. Each message must have a `topic` field (set by the per-topic processor) and a `data` field containing the Salesforce record fields. The `topic` is used to look up the correct `topic_mappings` entry which defines the sObject, operation, and write mode. **Realtime mode** uses the sObject Collections REST API (synchronous, up to 200 records/call). **Bulk mode** uses the Bulk API 2.0 (asynchronous, polls until complete). #### Common ```yml outputs: label: "" salesforce_sink: org_url: "" # No default (required) client_id: "" # No default (required) client_secret: "" # No default (required) restapi_version: v65.0 request_timeout: 30s max_retries: 10 bulk_batch_size: 1000 max_concurrent_bulk_jobs: 10 bulk_poll_interval: 5s batch_period: 5s max_in_flight: 1 topic_mappings: [] # No default (required) ``` #### Advanced ```yml outputs: label: "" salesforce_sink: org_url: "" # No default (required) client_id: "" # No default (required) client_secret: "" # No default (required) restapi_version: v65.0 request_timeout: 30s max_retries: 10 bulk_batch_size: 1000 max_concurrent_bulk_jobs: 10 bulk_poll_interval: 5s batch_period: 5s max_in_flight: 1 topic_mappings: [] # No default (required) ``` ## [](#fields)Fields ### [](#batch_period)`batch_period` Maximum period to wait before flushing an incomplete batch. **Type**: `string` **Default**: `5s` ### [](#bulk_batch_size)`bulk_batch_size` Number of records per bulk job. Also controls the output batch size. **Type**: `int` **Default**: `1000` ### [](#bulk_poll_interval)`bulk_poll_interval` How often to poll Salesforce for bulk job completion status. **Type**: `string` **Default**: `5s` ### [](#client_id)`client_id` Client ID for the Salesforce Connected App. **Type**: `string` ### [](#client_secret)`client_secret` Client secret for the Salesforce Connected App. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#max_concurrent_bulk_jobs)`max_concurrent_bulk_jobs` Maximum number of bulk jobs polling concurrently in the background. Each in-flight job buffers its CSV payload in memory. Lower this value if memory usage is a concern. **Type**: `int` **Default**: `10` ### [](#max_in_flight)`max_in_flight` Maximum number of batches to send concurrently. Increasing this value improves real-time write throughput. **Type**: `int` **Default**: `1` ### [](#max_retries)`max_retries` Maximum number of retries on a 429 Too Many Requests error. **Type**: `int` **Default**: `10` ### [](#org_url)`org_url` Salesforce instance base URL (for example, [https://your-domain.salesforce.com](https://your-domain.salesforce.com)). **Type**: `string` ### [](#request_timeout)`request_timeout` Timeout for HTTP requests. **Type**: `string` **Default**: `30s` ### [](#restapi_version)`restapi_version` Salesforce REST API version to use (for example, v65.0). **Type**: `string` **Default**: `v65.0` ### [](#topic_mappings)`topic_mappings[]` Per-topic Salesforce write configuration. Each entry maps a Kafka topic to an sObject and write settings. **Type**: `object` ### [](#topic_mappings-all_or_none)`topic_mappings[].all_or_none` Real-time only: rolls back the entire batch if any record fails. **Type**: `bool` **Default**: `false` ### [](#topic_mappings-external_id_field)`topic_mappings[].external_id_field` External ID field name. Required for upsert operations. **Type**: `string` **Default**: `""` ### [](#topic_mappings-mode)`topic_mappings[].mode` Write mode: `realtime` (sObject Collections API) or `bulk` (Bulk API 2.0). **Type**: `string` **Default**: `realtime` ### [](#topic_mappings-operation)`topic_mappings[].operation` Write operation: insert, update, upsert, or delete. **Type**: `string` **Default**: `upsert` ### [](#topic_mappings-sobject)`topic_mappings[].sobject` Salesforce sObject API name (for example, Account, Contact, MyObject\_\_c). **Type**: `string` ### [](#topic_mappings-topic)`topic_mappings[].topic` Kafka topic name to match against the message’s `topic` field. **Type**: `string` --- # Page 189: schema_registry **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/schema_registry.md --- # schema\_registry --- title: schema_registry latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/schema_registry page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/schema_registry.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/schema_registry.adoc page-git-created-date: "2024-09-04" page-git-modified-date: "2026-01-05" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/schema_registry/)[Input](/redpanda-connect/components/inputs/schema_registry/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/schema_registry/ "View the Cloud version of this component"), Self-Managed Publishes schemas to a schema registry. This output uses the [Franz Kafka Schema Registry client](https://github.com/twmb/franz-go/tree/master/pkg/sr). Introduced in version 4.32.2. #### Common ```yml outputs: label: "" schema_registry: url: "" # No default (required) subject: "" # No default (required) max_in_flight: 64 ``` #### Advanced ```yml outputs: label: "" schema_registry: url: "" # No default (required) subject: "" # No default (required) subject_compatibility_level: "" # No default (optional) backfill_dependencies: true translate_ids: false normalize: true remove_metadata: true remove_rule_set: true input_resource: schema_registry_input tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] max_in_flight: 64 oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} ``` ## [](#performance)Performance The `schema_registry` output sends multiple messages in parallel for improved performance. You can use the `max_in_flight` field to tune the maximum number of in-flight messages, or message batches. ## [](#example)Example This example writes schemas to a schema registry instance and logs errors for existing schemas. ```yaml output: fallback: - schema_registry: url: http://localhost:8082 subject: ${! @schema_registry_subject } - switch: cases: - check: '@fallback_error == "request returned status: 422"' output: drop: {} processors: - log: message: | Subject '${! @schema_registry_subject }' version ${! @schema_registry_version } already has schema: ${! content() } - output: reject: ${! @fallback_error } ``` ## [](#fields)Fields ### [](#backfill_dependencies)`backfill_dependencies` Backfill missing schema references and previous schema versions. If set to `true`, you must also configure a [`schema_registry`](../../inputs/schema_registry/) input to read source schemas. **Type**: `bool` **Default**: `true` ### [](#basic_auth)`basic_auth` Configure basic authentication for requests from this component to your schema registry. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` The password to use for authentication. Used together with `username` for basic authentication or with encrypted private keys for secure access. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` The username of the account credentials to authenticate as. Used together with `password` for basic authentication. **Type**: `string` **Default**: `""` ### [](#input_resource)`input_resource` The label of the [`schema_registry` input](../../inputs/schema_registry/) from which to read source schemas. **Type**: `string` **Default**: `schema_registry_input` ### [](#jwt)`jwt` Beta Configure JSON Web Token (JWT) authentication for secure data transmission from this component to your schema registry. This feature is in beta and may change in future releases. **Type**: `object` ### [](#jwt-claims)`jwt.claims` Values used to pass the identity of the authenticated entity to the service provider. In this case, between this component and the schema registry. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` The key/value pairs that identify the type of token and signing algorithm. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` A PEM-encoded file containing a private key that is formatted using either PKCS1 or PKCS8 standards. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` The method used to sign the token, such as RS256, RS384, RS512 or EdDSA. **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this number to improve throughput. **Type**: `int` **Default**: `64` ### [](#normalize)`normalize` Normalize schemas. Requires version 4.61.0 or later. **Type**: `bool` **Default**: `true` ### [](#oauth)`oauth` Configure OAuth version 1.0 to give this component authorized access to your schema registry. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` The value this component can use to gain access to the schema registry. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` The secret that establishes ownership of the `oauth.access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` The value used to identify this component or client to your schema registry. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#remove_metadata)`remove_metadata` Removes metadata fields from schema output. Use this to produce leaner schema definitions for downstream consumers or when metadata is not required. Requires version 4.61.0 or later. **Type**: `bool` **Default**: `true` ### [](#remove_rule_set)`remove_rule_set` Removes rule set definitions from schema output. Useful for simplifying schemas when rule sets are not required by consumers or applications. Requires version 4.61.0 or later. **Type**: `bool` **Default**: `true` ### [](#subject)`subject` The subject name. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#subject_compatibility_level)`subject_compatibility_level` The compatibility level for the subject. Can be one of `BACKWARD`, `BACKWARD_TRANSITIVE`, `FORWARD`, `FORWARD_TRANSITIVE`, `FULL`, `FULL_TRANSITIVE`, `NONE`. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#translate_ids)`translate_ids` When set to `true`, this field automatically translates the schema ID in each message to match the corresponding schema in the destination schema registry. The updated message is then written to the destination schema registry. **Type**: `bool` **Default**: `false` ### [](#url)`url` The base URL of the schema registry service. **Type**: `string` --- # Page 190: sftp **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/sftp.md --- # sftp --- title: sftp latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/sftp page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/sftp.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/sftp.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/sftp/)[Input](/redpanda-connect/components/inputs/sftp/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/sftp/ "View the Cloud version of this component"), Self-Managed Writes files to an SFTP server. Introduced in version 3.39.0. #### Common ```yml outputs: label: "" sftp: address: "" # No default (required) credentials: username: "" password: "" host_public_key_file: "" # No default (optional) host_public_key: "" # No default (optional) private_key_file: "" # No default (optional) private_key: "" # No default (optional) private_key_pass: "" path: "" # No default (required) codec: all-bytes max_in_flight: 64 ``` #### Advanced ```yml outputs: label: "" sftp: address: "" # No default (required) connection_timeout: 30s credentials: username: "" password: "" host_public_key_file: "" # No default (optional) host_public_key: "" # No default (optional) private_key_file: "" # No default (optional) private_key: "" # No default (optional) private_key_pass: "" path: "" # No default (required) codec: all-bytes max_in_flight: 64 ``` In order to have a different path for each object you should use function interpolations described [here](../../../configuration/interpolation/#bloblang-queries). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. ## [](#fields)Fields ### [](#address)`address` The address (hostname or IP address) of the SFTP server to connect to. **Type**: `string` ### [](#codec)`codec` The way in which the bytes of messages should be written out into the output data stream. It’s possible to write lines using a custom delimiter with the `delim:x` codec, where x is the character sequence custom delimiter. **Type**: `string` **Default**: `all-bytes` | Option | Summary | | --- | --- | | all-bytes | Only applicable to file based outputs. Writes each message to a file in full, if the file already exists the old content is deleted. | | append | Append each message to the output stream without any delimiter or special encoding. | | delim:x | Append each message to the output stream followed by a custom delimiter. | | lines | Append each message to the output stream followed by a line break. | ```yaml # Examples: codec: lines # --- codec: delim: # --- codec: delim:foobar ``` ### [](#connection_timeout)`connection_timeout` The connection timeout to use when connecting to the target server. **Type**: `string` **Default**: `30s` ### [](#credentials)`credentials` The credentials required to log in to the SFTP server. This can include a username and password, or a private key for secure access. **Type**: `object` ### [](#credentials-host_public_key)`credentials.host_public_key` The raw contents of the SFTP server’s public key, used for host key verification. **Type**: `string` ### [](#credentials-host_public_key_file)`credentials.host_public_key_file` The path to the SFTP server’s public key file, used for host key verification. **Type**: `string` ### [](#credentials-password)`credentials.password` The password to use for authentication. Used together with `username` for basic authentication or with encrypted private keys for secure access. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#credentials-private_key)`credentials.private_key` The private key used to authenticate with the SFTP server. This field provides an alternative to the [`private_key_file`](#credentials-private_key_file). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#credentials-private_key_file)`credentials.private_key_file` The path to a private key file used to authenticate with the SFTP server. You can also provide a private key using the [`private_key`](#credentials-private_key) field. **Type**: `string` ### [](#credentials-private_key_pass)`credentials.private_key_pass` A passphrase for private key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#credentials-username)`credentials.username` The username required to authenticate with the SFTP server. **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#path)`path` The file to save the messages to on the SFTP server. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 191: slack_post **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/slack_post.md --- # slack\_post --- title: slack_post latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/slack_post page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/slack_post.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/slack_post.adoc page-git-created-date: "2025-05-02" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/slack_post/ "View the Cloud version of this component"), Self-Managed **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Posts a new message to a Slack channel using the Slack API method [chat.postMessage](https://api.slack.com/methods/chat.postMessage). Introduced in version 4.52.0. ```yml # Common configuration fields, showing default values output: label: "" slack_post: bot_token: "" # No default (required) channel_id: "" # No default (required) thread_ts: "" # No default (optional) text: "" # No default (optional) blocks: "" # No default (optional) markdown: true unfurl_links: false unfurl_media: true link_names: 0 ``` See also: [Examples](#examples) ## [](#fields)Fields ### [](#blocks)`blocks` A Bloblang query that should return a JSON array of [Slack blocks](https://api.slack.com/reference/block-kit/blocks). You can either specify message content in the `text` or `blocks` fields, but not both. **Type**: `string` ### [](#bot_token)`bot_token` Your Slack bot user’s OAuth token, which must have the correct permissions to post messages to the target Slack channel. **Type**: `string` ### [](#channel_id)`channel_id` The encoded ID of the target Slack channel. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#link_names)`link_names` When set to `1`, this output finds and links to [user groups](https://api.slack.com/reference/surfaces/formatting#mentioning-groups) mentioned in Slack messages. **Type**: `bool` **Default**: `false` ### [](#markdown)`markdown` When set to `true`, this output accepts message content in Markdown format. **Type**: `bool` **Default**: `true` ### [](#text)`text` The text content of the message. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). You can either specify message content in the `text` or `blocks` fields, but not both. **Type**: `string` **Default**: `""` ### [](#thread_ts)`thread_ts` Specify the thread timestamp (`ts` value) of another message to post a reply within the same thread. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#unfurl_links)`unfurl_links` When set to `true`, this output provides previews of linked content in Slack messages. For more information about unfurling links, see the [Slack documentation](https://api.slack.com/reference/messaging/link-unfurling). **Type**: `bool` **Default**: `false` ### [](#unfurl_media)`unfurl_media` When set to `true`, this output provides previews of rich content in Slack messages, such as videos or embedded tweets. **Type**: `bool` **Default**: `true` ## [](#examples)Examples ### [](#echo-slackbot)Echo Slackbot A slackbot that echo messages from other users ```yaml input: slack: app_token: "${APP_TOKEN:xapp-demo}" bot_token: "${BOT_TOKEN:xoxb-demo}" pipeline: processors: - mutation: | # ignore hidden or non message events if this.event.type != "message" || (this.event.hidden | false) { root = deleted() } # Don't respond to our own messages if this.authorizations.any(auth -> auth.user_id == this.event.user) { root = deleted() } output: slack_post: bot_token: "${BOT_TOKEN:xoxb-demo}" channel_id: "${!this.event.channel}" thread_ts: "${!this.event.ts}" text: "ECHO: ${!this.event.text}" ``` --- # Page 192: slack_reaction **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/slack_reaction.md --- # slack\_reaction --- title: slack_reaction latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/slack_reaction page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/slack_reaction.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/slack_reaction.adoc description: Add or remove an emoji reaction to a Slack message. page-git-created-date: "2025-06-25" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/slack_reaction/ "View the Cloud version of this component"), Self-Managed **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Add or remove an emoji reaction to a Slack message using [`reactions.add`](https://api.slack.com/methods/reactions.add) and [`reactions.remove`](https://api.slack.com/methods/reactions.remove). ```yaml output: label: "" slack_reaction: bot_token: "" # No default (required) channel_id: "" # No default (required) timestamp: "" # No default (required) emoji: "" # No default (required) action: add max_in_flight: 64 ``` ## [](#fields)Fields ### [](#action)`action` Whether to add or remove the reaction. When set to `add`, the specified emoji reaction is applied to the target message. When set to `remove`, the emoji reaction is removed from the target message. **Type**: `string` **Default**: `add` **Options**: `add`, `remove` ### [](#bot_token)`bot_token` Your Slack Bot User OAuth token used to authenticate the API request. This token must have the necessary `reactions:write` and `channels:read` (or related) scopes. It typically begins with `xoxb-`. **Type**: `string` ### [](#channel_id)`channel_id` The unique Slack channel ID where the target message resides. Channel IDs usually start with `C` for public channels or `G` for private channels. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#emoji)`emoji` The name of the emoji to be added or removed, without surrounding colons. Use the plain emoji name, such as `thumbsup` or `tada`. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increasing this value can improve throughput in high-volume scenarios, but be cautious not to exceed Slack’s API rate limits. **Type**: `int` **Default**: `64` ### [](#timestamp)`timestamp` The timestamp of the message to react to. This is a unique identifier for the message, usually obtained from a previous Slack API call (such as `chat.postMessage` or `conversations.history`). It typically looks like a Unix timestamp with a decimal. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 193: snowflake_put **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/snowflake_put.md --- # snowflake\_put --- title: snowflake_put latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/snowflake_put page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/snowflake_put.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/snowflake_put.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/snowflake_put/ "View the Cloud version of this component"), Self-Managed **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. > 💡 **TIP** > > Use the [`snowflake_streaming` output](../snowflake_streaming/) for improved performance, cost-effectiveness, and ease of use. Sends messages to Snowflake stages and, optionally, calls Snowpipe to load this data into one or more tables. Introduced in version 4.0.0. #### Common ```yml outputs: label: "" snowflake_put: account: "" # No default (required) region: "" # No default (optional) cloud: "" # No default (optional) user: "" # No default (required) password: "" # No default (optional) private_key: "" # No default (optional) private_key_file: "" # No default (optional) private_key_pass: "" # No default (optional) role: "" # No default (required) database: "" # No default (required) warehouse: "" # No default (required) schema: "" # No default (required) stage: "" # No default (required) path: "" file_name: "" file_extension: "" compression: AUTO request_id: "" snowpipe: "" # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_in_flight: 1 ``` #### Advanced ```yml outputs: label: "" snowflake_put: account: "" # No default (required) region: "" # No default (optional) cloud: "" # No default (optional) user: "" # No default (required) password: "" # No default (optional) private_key: "" # No default (optional) private_key_file: "" # No default (optional) private_key_pass: "" # No default (optional) role: "" # No default (required) database: "" # No default (required) warehouse: "" # No default (required) schema: "" # No default (required) stage: "" # No default (required) path: "" file_name: "" file_extension: "" upload_parallel_threads: 4 compression: AUTO request_id: "" snowpipe: "" # No default (optional) client_session_keep_alive: false batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_in_flight: 1 ``` In order to use a different stage and / or Snowpipe for each message, you can use function interpolations as described in [Bloblang queries](../../../configuration/interpolation/#bloblang-queries). When using batching, messages are grouped by the calculated stage and Snowpipe and are streamed to individual files in their corresponding stage and, optionally, a Snowpipe `insertFiles` REST API call will be made for each individual file. ## [](#credentials)Credentials Two authentication mechanisms are supported: - User/password - Key Pair Authentication ### [](#userpassword)User/password This is a basic authentication mechanism which allows you to PUT data into a stage. However, it is not compatible with Snowpipe. ### [](#key-pair-authentication)Key pair authentication This authentication mechanism allows Snowpipe functionality, but it does require configuring an SSH Private Key beforehand. Please consult the [documentation](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication) for details on how to set it up and assign the Public Key to your user. Note that the Snowflake documentation [used to suggest](https://twitter.com/felipehoffa/status/1560811785606684672) using this command: ```bash openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out rsa_key.p8 ``` to generate an encrypted SSH private key. However, in this case, it uses an encryption algorithm called `pbeWithMD5AndDES-CBC`, which is part of the PKCS#5 v1.5 and is considered insecure. Due to this, Redpanda Connect does not support it and, if you wish to use password-protected keys directly, you must use PKCS#5 v2.0 to encrypt them by using the following command (as the current Snowflake docs suggest): ```bash openssl genrsa 2048 | openssl pkcs8 -topk8 -v2 des3 -inform PEM -out rsa_key.p8 ``` If you have an existing key encrypted with PKCS#5 v1.5, you can re-encrypt it with PKCS#5 v2.0 using this command: ```bash openssl pkcs8 -in rsa_key_original.p8 -topk8 -v2 des3 -out rsa_key.p8 ``` Please consult the [pkcs8 command documentation](https://linux.die.net/man/1/pkcs8) for details on PKCS#5 algorithms. ## [](#batching)Batching It’s common to want to upload messages to Snowflake as batched archives. The easiest way to do this is to batch your messages at the output level and join the batch of messages with an [`archive`](../../processors/archive/) and/or [`compress`](../../processors/compress/) processor. For the optimal batch size, please consult the Snowflake [documentation](https://docs.snowflake.com/en/user-guide/data-load-considerations-prepare.html). ## [](#snowpipe)Snowpipe Given a table called `BENTHOS_TBL` with one column of type `variant`: ```sql CREATE OR REPLACE TABLE BENTHOS_DB.PUBLIC.BENTHOS_TBL(RECORD variant) ``` and the following `BENTHOS_PIPE` Snowpipe: ```sql CREATE OR REPLACE PIPE BENTHOS_DB.PUBLIC.BENTHOS_PIPE AUTO_INGEST = FALSE AS COPY INTO BENTHOS_DB.PUBLIC.BENTHOS_TBL FROM (SELECT * FROM @%BENTHOS_TBL) FILE_FORMAT = (TYPE = JSON COMPRESSION = AUTO) ``` you can configure Redpanda Connect to use the implicit table stage `@%BENTHOS_TBL` as the `stage` and `BENTHOS_PIPE` as the `snowpipe`. In this case, you must set `compression` to `AUTO` and, if using message batching, you’ll need to configure an [`archive`](../../processors/archive/) processor with the `concatenate` format. Since the `compression` is set to `AUTO`, the [gosnowflake](https://github.com/snowflakedb/gosnowflake) client library will compress the messages automatically so you don’t need to add a [`compress`](../../processors/compress/) processor for message batches. If you add `STRIP_OUTER_ARRAY = TRUE` in your Snowpipe `FILE_FORMAT` definition, then you must use `json_array` instead of `concatenate` as the archive processor format. > 📝 **NOTE** > > Only Snowpipes with `FILE_FORMAT` `TYPE` `JSON` are currently supported. ## [](#snowpipe-troubleshooting)Snowpipe troubleshooting Snowpipe [provides](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-rest-apis.html) the `insertReport` and `loadHistoryScan` REST API endpoints which can be used to get information about recent Snowpipe calls. In order to query them, you’ll first need to generate a valid JWT token for your Snowflake account. There are two methods for doing so: - Using the `snowsql` [utility](https://docs.snowflake.com/en/user-guide/snowsql.html): ```bash snowsql --private-key-path rsa_key.p8 --generate-jwt -a -u ``` - Using the Python `sql-api-generate-jwt` [utility](https://docs.snowflake.com/en/developer-guide/sql-api/authenticating.html#generating-a-jwt-in-python): ```bash python3 sql-api-generate-jwt.py --private_key_file_path=rsa_key.p8 --account= --user= ``` Once you successfully generate a JWT token and store it into the `JWT_TOKEN` environment variable, then you can, for example, query the `insertReport` endpoint using `curl`: ```bash curl -H "Authorization: Bearer ${JWT_TOKEN}" "https://.snowflakecomputing.com/v1/data/pipes/../insertReport" ``` If you need to pass in a valid `requestId` to any of these Snowpipe REST API endpoints, you can set a [uuid\_v4()](../../../guides/bloblang/functions/#uuid_v4) string in a metadata field called `request_id`, log it via the [`log`](../../processors/log/) processor and then configure `request_id: ${ @request_id }` ). Alternatively, you can [enable debug logging](../../logger/about/) and Redpanda Connect will print the Request IDs that it sends to Snowpipe. ## [](#general-troubleshooting)General troubleshooting The underlying [`gosnowflake` driver](https://github.com/snowflakedb/gosnowflake) requires write access to the default directory to use for temporary files. Please consult the [`os.TempDir`](https://pkg.go.dev/os#TempDir) docs for details on how to change this directory via environment variables. A silent failure can occur due to [this issue](https://github.com/snowflakedb/gosnowflake/issues/701), where the underlying [`gosnowflake` driver](https://github.com/snowflakedb/gosnowflake) doesn’t return an error and doesn’t log a failure if it can’t figure out the current username. One way to trigger this behavior is by running Redpanda Connect in a Docker container with a non-existent user ID (such as `--user 1000:1000`). ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](../../../configuration/batching/). ## [](#examples)Examples ### [](#kafka-realtime-brokers)Kafka / realtime brokers Upload message batches from realtime brokers such as Kafka persisting the batch partition and offsets in the stage path and filename similarly to the [Kafka Connector scheme](https://docs.snowflake.com/en/user-guide/kafka-connector-ts.html#step-1-view-the-copy-history-for-the-table) and call Snowpipe to load them into a table. When batching is configured at the input level, it is done per-partition. ```yaml input: redpanda: seed_brokers: - localhost:9092 topics: - foo consumer_group: rpcn max_yield_batch_bytes: 8MB processors: - mapping: | meta kafka_start_offset = meta("kafka_offset").from(0) meta kafka_end_offset = meta("kafka_offset").from(-1) meta batch_timestamp = if batch_index() == 0 { now() } - mapping: | meta batch_timestamp = if batch_index() != 0 { meta("batch_timestamp").from(0) } output: snowflake_put: account: benthos user: test@benthos.dev private_key_file: path_to_ssh_key.pem role: ACCOUNTADMIN database: BENTHOS_DB warehouse: COMPUTE_WH schema: PUBLIC stage: "@%BENTHOS_TBL" path: benthos/BENTHOS_TBL/${! @kafka_partition } file_name: ${! @kafka_start_offset }_${! @kafka_end_offset }_${! meta("batch_timestamp") } upload_parallel_threads: 4 compression: NONE snowpipe: BENTHOS_PIPE ``` ### [](#no-compression)No compression Upload concatenated messages into a `.json` file to a table stage without calling Snowpipe. ```yaml output: snowflake_put: account: benthos user: test@benthos.dev private_key_file: path_to_ssh_key.pem role: ACCOUNTADMIN database: BENTHOS_DB warehouse: COMPUTE_WH schema: PUBLIC stage: "@%BENTHOS_TBL" path: benthos upload_parallel_threads: 4 compression: NONE batching: count: 10 period: 3s processors: - archive: format: concatenate ``` ### [](#parquet-format-with-snappy-compression)Parquet format with snappy compression Upload concatenated messages into a `.parquet` file to a table stage without calling Snowpipe. ```yaml output: snowflake_put: account: benthos user: test@benthos.dev private_key_file: path_to_ssh_key.pem role: ACCOUNTADMIN database: BENTHOS_DB warehouse: COMPUTE_WH schema: PUBLIC stage: "@%BENTHOS_TBL" path: benthos file_extension: parquet upload_parallel_threads: 4 compression: NONE batching: count: 10 period: 3s processors: - parquet_encode: schema: - name: ID type: INT64 - name: CONTENT type: BYTE_ARRAY default_compression: snappy ``` ### [](#automatic-compression)Automatic compression Upload concatenated messages compressed automatically into a `.gz` archive file to a table stage without calling Snowpipe. ```yaml output: snowflake_put: account: benthos user: test@benthos.dev private_key_file: path_to_ssh_key.pem role: ACCOUNTADMIN database: BENTHOS_DB warehouse: COMPUTE_WH schema: PUBLIC stage: "@%BENTHOS_TBL" path: benthos upload_parallel_threads: 4 compression: AUTO batching: count: 10 period: 3s processors: - archive: format: concatenate ``` ### [](#deflate-compression)DEFLATE compression Upload concatenated messages compressed into a `.deflate` archive file to a table stage and call Snowpipe to load them into a table. ```yaml output: snowflake_put: account: benthos user: test@benthos.dev private_key_file: path_to_ssh_key.pem role: ACCOUNTADMIN database: BENTHOS_DB warehouse: COMPUTE_WH schema: PUBLIC stage: "@%BENTHOS_TBL" path: benthos upload_parallel_threads: 4 compression: DEFLATE snowpipe: BENTHOS_PIPE batching: count: 10 period: 3s processors: - archive: format: concatenate - mapping: | root = content().compress("zlib") ``` ### [](#raw_deflate-compression)RAW\_DEFLATE compression Upload concatenated messages compressed into a `.raw_deflate` archive file to a table stage and call Snowpipe to load them into a table. ```yaml output: snowflake_put: account: benthos user: test@benthos.dev private_key_file: path_to_ssh_key.pem role: ACCOUNTADMIN database: BENTHOS_DB warehouse: COMPUTE_WH schema: PUBLIC stage: "@%BENTHOS_TBL" path: benthos upload_parallel_threads: 4 compression: RAW_DEFLATE snowpipe: BENTHOS_PIPE batching: count: 10 period: 3s processors: - archive: format: concatenate - mapping: | root = content().compress("flate") ``` ## [](#fields)Fields ### [](#account)`account` Account name, which is the same as the [Account Identifier](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html#where-are-account-identifiers-used). However, when using an [Account Locator](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html#using-an-account-locator-as-an-identifier), the Account Identifier is formatted as `..` and this field needs to be populated using the `` part. **Type**: `string` ### [](#batching-2)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#client_session_keep_alive)`client_session_keep_alive` Enable Snowflake keepalive mechanism to prevent the client session from expiring after 4 hours (error 390114). **Type**: `bool` **Default**: `false` ### [](#cloud)`cloud` Optional cloud platform field which needs to be populated when using an [Account Locator](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html#using-an-account-locator-as-an-identifier) and it must be set to the `` part of the Account Identifier (`..`). **Type**: `string` ```yaml # Examples: cloud: aws # --- cloud: gcp # --- cloud: azure ``` ### [](#compression)`compression` Compression type. **Type**: `string` **Default**: `AUTO` | Option | Summary | | --- | --- | | AUTO | Compression (gzip) is applied automatically by the output and messages must contain plain-text JSON. Default file_extension: gz. | | DEFLATE | Messages must be pre-compressed using the zlib algorithm (with zlib header, RFC1950). Default file_extension: deflate. | | GZIP | Messages must be pre-compressed using the gzip algorithm. Default file_extension: gz. | | NONE | No compression is applied and messages must contain plain-text JSON. Default file_extension: json. | | RAW_DEFLATE | Messages must be pre-compressed using the flate algorithm (without header, RFC1951). Default file_extension: raw_deflate. | | ZSTD | Messages must be pre-compressed using the Zstandard algorithm. Default file_extension: zst. | ### [](#database)`database` Database. **Type**: `string` ### [](#file_extension)`file_extension` Stage file extension. Will be derived from the configured `compression` if not set or empty. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). Requires version v4.12.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: file_extension: csv # --- file_extension: parquet ``` ### [](#file_name)`file_name` Stage file name. Will be equal to the Request ID if not set or empty. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). Requires version v4.12.0 or later. **Type**: `string` **Default**: `""` ### [](#max_in_flight)`max_in_flight` The maximum number of parallel message batches to have in flight at any given time. **Type**: `int` **Default**: `1` ### [](#password)`password` An optional password. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#path)`path` Stage path. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#private_key)`private_key` Your private SSH key. When using encrypted keys, you must also set a value for [`private_key_pass`](#private_key_pass). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#private_key_file)`private_key_file` The path to a file containing your private SSH key. When using encrypted keys, you must also set a value for [`private_key_pass`](#private_key_pass). **Type**: `string` ### [](#private_key_pass)`private_key_pass` The passphrase for your private SSH key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#region)`region` Optional region field which needs to be populated when using an [Account Locator](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html#using-an-account-locator-as-an-identifier) and it must be set to the `` part of the Account Identifier (`..`). **Type**: `string` ```yaml # Examples: region: us-west-2 ``` ### [](#request_id)`request_id` Request ID. Will be assigned a random UUID (v4) string if not set or empty. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). Requires version v4.12.0 or later. **Type**: `string` **Default**: `""` ### [](#role)`role` Role. **Type**: `string` ### [](#schema)`schema` Schema. **Type**: `string` ### [](#snowpipe-2)`snowpipe` An optional Snowpipe name. Use the `` part from `..`. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#stage)`stage` Stage name. Use either one of the [supported](https://docs.snowflake.com/en/user-guide/data-load-local-file-system-create-stage.html) stage types. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#upload_parallel_threads)`upload_parallel_threads` Specifies the number of threads to use for uploading files. **Type**: `int` **Default**: `4` ### [](#user)`user` Username. **Type**: `string` ### [](#warehouse)`warehouse` Warehouse. **Type**: `string` --- # Page 194: snowflake_streaming **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/snowflake_streaming.md --- # snowflake\_streaming --- title: snowflake_streaming latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/snowflake_streaming page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/snowflake_streaming.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/snowflake_streaming.adoc page-git-created-date: "2024-11-19" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/snowflake_streaming/ "View the Cloud version of this component"), Self-Managed **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Allows Snowflake to ingest data from your data pipeline using [Snowpipe Streaming](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-overview). To help you configure your own `snowflake_streaming` output, this page includes [example data pipelines](#example-pipelines). Introduced in version 4.39.0. #### Common ```yml outputs: label: "" snowflake_streaming: account: "" # No default (required) user: "" # No default (required) role: "" # No default (required) database: "" # No default (required) schema: "" # No default (required) table: "" # No default (required) private_key: "" # No default (optional) private_key_file: "" # No default (optional) private_key_pass: "" # No default (optional) mapping: "" # No default (optional) init_statement: "" # No default (optional) schema_evolution: enabled: "" # No default (required) ignore_nulls: true processors: [] # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_in_flight: 4 ``` #### Advanced ```yml outputs: label: "" snowflake_streaming: account: "" # No default (required) url: "" # No default (optional) user: "" # No default (required) role: "" # No default (required) database: "" # No default (required) schema: "" # No default (required) table: "" # No default (required) private_key: "" # No default (optional) private_key_file: "" # No default (optional) private_key_pass: "" # No default (optional) mapping: "" # No default (optional) init_statement: "" # No default (optional) schema_evolution: enabled: "" # No default (required) ignore_nulls: true processors: [] # No default (optional) build_options: parallelism: 1 chunk_size: 50000 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) max_in_flight: 4 channel_prefix: "" # No default (optional) channel_name: "" # No default (optional) offset_token: "" # No default (optional) commit_backoff: initial_interval: 32ms max_interval: 512ms max_elapsed_time: 60s multiplier: 2 message_format: object timestamp_format: 2006-01-02T15:04:05.999999999Z07:00 ``` ## [](#conversion-of-message-data-into-snowflake-table-rows)Conversion of message data into Snowflake table rows Message data conversion to Snowflake table rows is determined by the: - Output message contents. - [Schema evolution settings](#schema_evolution). - Schema of the [target Snowflake table](#table). The following scenarios highlight how these three factors affect data written to the target table. > 📝 **NOTE** > > For reduced complexity, consider [turning on schema evolution](#schema_evolution), which automatically creates and updates the Snowflake table schema based on message contents. ### [](#scenario-data-and-table-schema-match-schema-evolution-turned-on-or-off)Scenario: Data and table schema match (schema evolution turned on or off) An output message matches the existing table schema, and the `schema_evolution.enabled` field is set to `true` or `false`. The target Snowflake table has two columns: - `product_id` (NUMBER) - `product_code` (STRING) A pipeline generates the following message: ```json {"product_id": 521, "product_code": “EST-PR”} ``` In this scenario: - The JSON keys in the message (`"product_id"` and `"product_code"`) match column names in the target Snowflake table. - The message values match the column data types. (If there was a data mismatch, the message would be rejected.) - Redpanda Connect inserts the message values into a new row in the target Snowflake table. | product_id | product_code | | --- | --- | | 521 | EST-PR | ### [](#scenario-data-and-table-schema-mismatch-schema-evolution-turned-on)Scenario: Data and table schema mismatch (schema evolution turned on) An output message includes schema updates, and the `schema_evolution.enabled` field is set to `true`. The target Snowflake table has the same two columns as the [previous scenario](#scenario-data-and-table-schema-match-schema-evolution-turned-on-or-off): - `product_id` (NUMBER) - `product_code` (STRING) This time, the pipeline generates the following message: ```json {"product_batch": 11111, "product_color": “yellow”} ``` In this scenario: - The JSON keys (`"product_batch"` and `"product_color"`) do not match column names in the target Snowflake table. - As schema evolution is enabled, Redpanda Connect adds two new columns to the target table with data types derived from the output message values. For more information about the mapping of data types, see [Supported data formats for Snowflake columns](#supported-data-formats-for-snowflake-columns). - Redpanda Connect inserts the message values into a new table row. | product_id | product_code | product_batch | product_color | | --- | --- | --- | --- | | (null) | (null) | 11111 | yellow | > 📝 **NOTE** > > You can [configure processors](#schema_evolution-processors) to override the schema updates derived from the message values. ### [](#scenario-data-and-table-schema-mismatch-schema-evolution-turned-off)Scenario: Data and table schema mismatch (schema evolution turned off) An output message includes schema updates, and the `schema_evolution.enabled` field is set to `false`. The target Snowflake table has the same two columns: - `product_id` (NUMBER) - `product_code` (STRING) The pipeline generates the same message as the [previous scenario](#scenario-data-and-table-schema-mismatch-schema-evolution-turned-on): ```json {"product_batch": 11111, "product_color": “yellow”} ``` In this scenario: - The JSON keys (`"product_batch"` and `"product_color"`) do not match any existing column names. - Because schema evolution is turned off, Redpanda Connect ignores the extra column names and values and inserts a row of null values. | product_id | product_code | | --- | --- | | (null) | (null) | ## [](#supported-data-formats-for-snowflake-columns)Supported data formats for Snowflake columns The message data from your output must match the columns in the Snowflake table that you want to write data to. The following table shows you the [column data types supported by Snowflake](https://docs.snowflake.com/en/sql-reference/intro-summary-data-types) and how they correspond to the [Bloblang data types](../../../guides/bloblang/methods/#type) in Redpanda Connect. | Snowflake column data type | Bloblang data types | | --- | --- | | CHAR, VARCHAR | string | | BINARY | string or bytes | | NUMBER | number, or string where the string is parsed into a number | | FLOAT, including special values, such as NaN (Not a Number), -inf (negative infinity), and inf (positive infinity) | number | | BOOLEAN | bool, or number where a non-zero number is true | | TIME, DATE, TIMESTAMP | timestamp, or number where the number is a converted to a Unix timestamp, or string where the string is parsed using RFC 3339 format | | VARIANT, ARRAY, OBJECT | Any data type converted into JSON | | GEOGRAPHY,GEOMETRY | Not supported | ## [](#authentication)Authentication You can authenticate with Snowflake using an [RSA key pair](https://docs.snowflake.com/en/user-guide/key-pair-auth). Either specify: - A PEM-encoded private key, in the [`private_key` field](#private_key). - The path to a file from which the output can load the private RSA key, in the [`private_key_file` field](#private_key_file). ## [](#performance)Performance For improved performance, this output: - Sends multiple messages in parallel. You can tune the maximum number of in-flight messages (or message batches) with the field `max_in_flight`. - Sends messages as a batch. You can configure batches at both the input and output level. For more information, see [Message Batching](../../../configuration/batching/). ### [](#batch-sizes)Batch sizes Redpanda recommends that every message batch writes at least 16 MiB of compressed output to Snowflake. You can monitor batch sizes using the `snowflake_compressed_output_size_bytes` metric. ### [](#metrics)Metrics This output emits the following metrics. | Metric name | Description | | --- | --- | | snowflake_compressed_output_size_bytes | The size in bytes of each message batch uploaded to Snowflake. | | snowflake_convert_latency_ns | The time taken to convert messages into the Snowflake column data types. | | snowflake_serialize_latency_ns | The time taken to serialize the converted columnar data into a file for upload to Snowflake. | | snowflake_build_output_latency_ns | The time taken to build the file that is uploaded to Snowflake. This metric is the sum of snowflake_convert_latency_ns + snowflake_serialize_latency_ns. | | snowflake_upload_latency_ns | The time taken to upload the output file to Snowflake. | | snowflake_register_latency_ns | The time taken to register the uploaded output file with Snowflake. | | snowflake_commit_latency_ns | The time taken to commit the uploaded data updates to the target Snowflake table. | ## [](#fields)Fields ### [](#account)`account` The [Snowflake account name to use](https://docs.snowflake.com/en/user-guide/admin-account-identifier#account-name). Use the format `-` where: - The `` is the name of your Snowflake organization. - The `` is the unique name of your account with your Snowflake organization. To find the correct value for this field, run the following query in Snowflake: ```sql WITH HOSTLIST AS (SELECT * FROM TABLE(FLATTEN(INPUT => PARSE_JSON(SYSTEM$allowlist())))) SELECT REPLACE(VALUE:host,'.snowflakecomputing.com','') AS ACCOUNT_IDENTIFIER FROM HOSTLIST WHERE VALUE:type = 'SNOWFLAKE_DEPLOYMENT_REGIONLESS'; ``` **Type**: `string` ```yaml # Examples: account: ORG-ACCOUNT ``` ### [](#batching)`batching` Lets you configure a [batching policy](../../../configuration/batching/). Type\*: `object` ```yml # Examples batching: byte_size: 5000 count: 0 period: 1s batching: count: 10 period: 1s batching: check: this.contains("END BATCH") count: 0 period: 1m ``` **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. All resulting messages are flushed as a single batch, and therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#build_options)`build_options` Options for optimizing the build of the output data that is sent to Snowflake. Monitor the `snowflake_build_output_latency_ns` metric to assess whether you need to update these options. **Type**: `object` ### [](#build_options-chunk_size)`build_options.chunk_size` The number of table rows to submit in each chunk for processing. **Type**: `int` **Default**: `50000` ### [](#build_options-parallelism)`build_options.parallelism` The maximum amount of parallel processing to use when building the output for Snowflake. **Type**: `int` **Default**: `1` ### [](#channel_name)`channel_name` The channel name to use when connecting to a Snowflake table. Duplicate channel names cause errors and prevent multiple instances of Redpanda Connect from writing at the same time, and so this field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). Redpanda Connect assumes that a message batch contains messages for a single channel, which means that interpolation is only executed on the first message in each batch. If your pipeline uses an input that is partitioned, such as an Apache Kafka topic, batch messages at the input level to make sure all messages are processed by the same channel. You can specify either the `channel_name` or `channel_prefix`, but not both. If neither field is populated, this output creates a channel name based on a table’s fully-qualified name, which results in a single stream per table. > 📝 **NOTE** > > Snowflake limits the number of streams per table to 10,000. If you need to use more than 10,000 streams, contact [Snowflake support](https://www.snowflake.com/en/support/). **Type**: `string` ```yaml # Examples: channel_name: partition-${!@kafka_partition} ``` ### [](#channel_prefix)`channel_prefix` The prefix to use when creating a channel name for connecting to a Snowflake table. Adding a `channel_prefix` avoids the creation of duplicate channel names, which result in errors and prevent multiple instances of Redpanda Connect from writing at the same time. You can specify either the `channel_prefix` or `channel_name`, but not both. If neither field is populated, this output creates a channel name based on a table’s fully-qualified name, which results in a single stream per table. The maximum number of channels open at any time is determined by the value in the `max_in_flight` field. > 📝 **NOTE** > > Snowflake limits the number of streams per table to 10,000. If you need to use more than 10,000 streams, contact [Snowflake support](https://www.snowflake.com/en/support/). **Type**: `string` ```yaml # Examples: channel_prefix: channel-${HOST} ``` ### [](#commit_backoff)`commit_backoff` Control how frequently Snowflake is polled to check if data has been committed. **Type**: `object` ### [](#commit_backoff-initial_interval)`commit_backoff.initial_interval` The initial period to wait between status polls. **Type**: `string` **Default**: `32ms` ### [](#commit_backoff-max_elapsed_time)`commit_backoff.max_elapsed_time` The maximum total time to wait for data to be committed. If zero then no limit is used. **Type**: `string` **Default**: `60s` ### [](#commit_backoff-max_interval)`commit_backoff.max_interval` The maximum period to wait between status polls. **Type**: `string` **Default**: `512ms` ### [](#commit_backoff-multiplier)`commit_backoff.multiplier` The factor by which the poll interval grows on each attempt. **Type**: `float` **Default**: `2` ### [](#database)`database` The Snowflake database you want to write data to. **Type**: `string` ```yaml # Examples: database: MY_DATABASE ``` ### [](#init_statement)`init_statement` Optional SQL statements to execute immediately after this output connects to Snowflake for the first time. This is a useful way to initialize tables before processing data. > 📝 **NOTE** > > Make sure your SQL statements are idempotent, so they do not cause issues when run multiple times after service restarts. **Type**: `string` ```yaml # Examples: init_statement: |- CREATE TABLE IF NOT EXISTS mytable (amount NUMBER); # --- init_statement: |- ALTER TABLE t1 ALTER COLUMN c1 DROP NOT NULL; ALTER TABLE t1 ADD COLUMN a2 NUMBER; ``` ### [](#mapping)`mapping` The [Bloblang `mapping`](../../../guides/bloblang/about/) to execute on each message. **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this number to improve throughput until performance plateaus. **Type**: `int` **Default**: `4` ### [](#message_format)`message_format` The format to expect incoming messages from the rest of the pipeline. **Type**: `string` **Default**: `object` | Option | Summary | | --- | --- | | array | Messages are an array of values where each position matches the ordinal of the column in Snowflake. | | object | Messages are JSON or Bloblang objects where each key is the Snowflake column name and the value is the column value. | ```yaml # Examples: message_format: array ``` ### [](#offset_token)`offset_token` The offset token to use for exactly-once delivery of data to a Snowflake table. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). This output assumes that messages within a batch are in increasing order by offset token. When data is sent on a channel, the offset token of each message in the batch is compared to the latest token processed by the channel. If the offset token is lexicographically less than the latest token, it’s assumed the message is a duplicate and is dropped. Messages must be delivered to the output in order, otherwise they are processed as duplicates and dropped. To avoid dropping retried messages if later messages have succeeded in the meantime, use a dead-letter queue to process failed messages. See the [Ingesting data exactly once from Redpanda](#example-pipelines) example. > 📝 **NOTE** > > If you’re using a numeric value as an offset token, pad the value so that it’s lexicographically ordered in its string representation because offset tokens are compared in string form. For more details, see the [Ingesting data exactly once from Redpanda](#example-pipelines) example. For more information about offset tokens, see [Snowflake Documentation](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-overview#offset-tokens). **Type**: `string` ```yaml # Examples: offset_token: offset-${!"%016X".format(@kafka_offset)} # --- offset_token: postgres-${!@lsn} ``` ### [](#private_key)`private_key` The PEM-encoded private RSA key to use for authentication with Snowflake. You must specify a value for this field or the `private_key_file` field. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#private_key_file)`private_key_file` A `.p8`, PEM-encoded file to load the private RSA key from. You must specify a value for this field or the `private_key` field. **Type**: `string` ### [](#private_key_pass)`private_key_pass` If the RSA key is encrypted, specify the RSA key passphrase. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#role)`role` The role of the user specified in the `user` field. The user’s role must have the [required privileges](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-overview#required-access-privileges) to call the Snowpipe Streaming APIs. For more information about user roles, see the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/admin-user-management#user-roles). **Type**: `string` ```yaml # Examples: role: ACCOUNTADMIN ``` ### [](#schema)`schema` The schema of the Snowflake database you want to write data to. **Type**: `string` ```yaml # Examples: schema: PUBLIC ``` ### [](#schema_evolution)`schema_evolution` Options to control schema updates when messages are written to the Snowflake table. **Type**: `object` ### [](#schema_evolution-enabled)`schema_evolution.enabled` Whether schema evolution is enabled. When set to `true`, the Snowflake table is automatically created based on the schema of the first message written to it, if the table does not already exist. As new fields are added to subsequent messages in the pipeline, new columns are created in the Snowflake table. Any required columns are marked as `nullable` if new messages do not include data for them. **Type**: `bool` ### [](#schema_evolution-ignore_nulls)`schema_evolution.ignore_nulls` When set to `true` and schema evolution is enabled, new columns that have `null` values _are not_ added to the Snowflake table. This behavior: - Prevents unnecessary schema changes caused by placeholder or incomplete data. - Avoids creating table columns with incorrect data types. > 📝 **NOTE** > > Redpanda does not recommend updating the default setting unless you are confident about the data type of `null` columns in advance. **Type**: `bool` **Default**: `true` ### [](#schema_evolution-processors)`schema_evolution.processors[]` A series of processors to execute when new columns are added to the Snowflake table. You can use these processors to: - Run side effects when the schema evolves. - Enrich the message with additional information to guide the schema changes. For example, a processor could read the schema from the schema registry that a message was produced with and use that schema to determine the data type of the new column in Snowflake. The input to these processors is an object with the value and name of the new column, the original message, and details of the Snowflake table the output writes to. For example: `{"value": 42.3, "name":"new_data_field", "message": {"existing_data_field": 42, "new_data_field": "db_field_name"}, "db": MY_DATABASE", "schema": "MY_SCHEMA", "table": "MY_TABLE"}` The output from the processors must be a valid message, which contains a string that specifies the column type for the new column in Snowflake. The metadata remains the same as in the original message that triggered the schema update. **Type**: `processor` ```yaml # Examples: processors: - mapping: |- root = match this.value.type() { this == "string" => "STRING" this == "bytes" => "BINARY" this == "number" => "DOUBLE" this == "bool" => "BOOLEAN" this == "timestamp" => "TIMESTAMP" _ => "VARIANT" } ``` ### [](#table)`table` The Snowflake table you want to write data to. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: table: MY_TABLE ``` ### [](#timestamp_format)`timestamp_format` The format to parse string values for `TIMESTAMP`, `TIMESTAMP_LTZ` and `TIMESTAMP_NTZ` columns. Should be a layout for [time.Parse](https://pkg.go.dev/time#Parse) in Go. **Type**: `string` **Default**: `2006-01-02T15:04:05.999999999Z07:00` ### [](#url)`url` Specify a custom URL to connect to Snowflake. This parameter overrides the default URL, which is automatically generated from the value of `output.snowflake_streaming.account`. By default, the URL is constructed as follows: `[https://.snowflakecomputing.com](https://.snowflakecomputing.com)`. **Type**: `string` ```yaml # Examples: url: https://org-account.privatelink.snowflakecomputing.com ``` ### [](#user)`user` Specify a user to run the Snowpipe Stream. To learn how to create a user, see the [Snowflake documentation](https://docs.snowflake.com/en/user-guide/admin-user-management). **Type**: `string` ## [](#example-pipelines)Example pipelines The following examples show you how to ingest, process, and write data to Snowflake from: - A PostgreSQL table using change data capture (CDC) - A Redpanda cluster - A REST API that posts JSON payloads to a HTTP server See also: [Ingest data into Snowflake cookbook](../../../cookbooks/snowflake_ingestion/) ### Write data exactly once to a Snowflake table using CDC Send data from a PostgreSQL table and write it to Snowflake exactly once using PostgreSQL logical replication. This example includes some important features: - To make sure that a Snowflake streaming channel does not assume that older data is already committed, the configuration sets a 45-second interval between message batches. This interval prevents a message batch from being sent while another batch is retried. - The log sequence number of each data update from the Write-Ahead Log (WAL) in PostgreSQL makes sure that data is only uploaded once to the `snowflake_streaming` output, and that messages sent to the output are already lexicographically ordered. > 📝 **NOTE** > > To do exactly-once data delivery, it’s important that records are delivered in order to the output, and are correctly partitioned. Before you start, read the [`offset_token`](#offset_token) field description. Alternatively, remove the `offset_token` field to use Redpanda Connect’s default at-least-once delivery model. ```yaml input: postgres_cdc: dsn: postgres://foouser:foopass@localhost:5432/foodb schema: "public" tables: ["my_pg_table"] # Use very large batches. Each batch is sent to Snowflake individually, # so to optimize query performance, use the largest file size # your memory allows batching: count: 50000 period: 45s # Set an interval between message batches to prevent multiple batches # from being in flight at once checkpoint_limit: 1 output: snowflake_streaming: # Using the log sequence number makes sure data is only updated exactly once offset_token: "${!@lsn}" # Sending a single ordered log means you can only send one update # at a time and properly increment the offset_token # and use only a single channel. max_in_flight: 1 account: "MYSNOW-ACCOUNT" user: MYUSER role: ACCOUNTADMIN database: "MYDATABASE" schema: "PUBLIC" table: "MY_PG_TABLE" private_key_file: "my/private/key.p8" ``` ### Ingest data exactly once from Redpanda Ingest data from Redpanda using consumer groups, decode the schema using the schema registry, then write the corresponding data into Snowflake. This example includes some important features: - To create multiple Redpanda Connect streams to write to each output table, you need a unique channel prefix per stream. The `channel_prefix` field constructs a unique prefix for each stream using the host name. - To prevent message failures from being retried and changing the order of delivered messages, a dead-letter queue processes them. For more details about message ordering, see the [`redpanda_common` input](../../inputs/redpanda_common/#ordering) documentation. > 📝 **NOTE** > > To do exactly-once data delivery, it’s important that records are delivered in order to the output, and are correctly partitioned. Before you start, read the [`channel_name`](#channel_name) and [`offset_token`](#offset_token) field descriptions. Alternatively, remove the `offset_token` field to use Redpanda Connect’s default at-least-once delivery model. ```yaml input: redpanda_common: topics: ["my_topic_going_to_snow"] consumer_group: "redpanda_connect_to_snowflake" # Use very large batches. Each batch is sent to Snowflake individually, # so to optimize query performance, use the largest file size # your memory allows fetch_max_bytes: 100MiB fetch_min_bytes: 50MiB partition_buffer_bytes: 100MiB pipeline: processors: - schema_registry_decode: url: "redpanda.example.com:8081" basic_auth: enabled: true username: MY_USER_NAME password: "${TODO}" output: fallback: - snowflake_streaming: # To write an ordered stream of messages, each partition in # Apache Kafka gets its own channel. channel_name: "partition-${!@kafka_partition}" # Offsets are lexicographically sorted in string form by padding with # leading zeros offset_token: offset-${!"%016X".format(@kafka_offset)} account: "MYSNOW-ACCOUNT" user: MYUSER role: ACCOUNTADMIN database: "MYDATABASE" schema: "PUBLIC" table: "MYTABLE" private_key_file: "my/private/key.p8" schema_evolution: enabled: true # To prevent delivery failures from changing the order of # delivered records, it's important that they are immediately # sent to a dead-letter queue. - retry: output: redpanda_common: topic: "dead_letter_queue" ``` ### HTTP server to push data to Snowflake Create a HTTP server input that receives HTTP PUT requests with JSON payloads. The payloads are buffered locally then written to Snowflake in batches. To create multiple Redpanda Connect streams to write to each output table, you need a unique channel prefix per stream. In this example, the `channel_prefix` field constructs a unique prefix for each stream using the host name. > 📝 **NOTE** > > Using a buffer to immediately respond to the HTTP requests may result in data loss if there are delivery failures between the output and Snowflake. For more information about the configuration of buffers, see [buffers](../../buffers/memory/). Alternatively, remove the buffer entirely to respond to the HTTP request only once the data is written to Snowflake. ```yaml input: http_server: path: /snowflake buffer: memory: # Max inflight data before applying backpressure limit: 524288000 # 50MiB # Batching policy the size of the files sent to Snowflake batch_policy: enabled: true byte_size: 33554432 # 32MiB period: "10s" output: snowflake_streaming: account: "MYSNOW-ACCOUNT" user: MYUSER role: ACCOUNTADMIN database: "MYDATABASE" schema: "PUBLIC" table: "MYTABLE" private_key_file: "my/private/key.p8" channel_prefix: "snowflake-channel-for-${HOST}" schema_evolution: enabled: true ``` --- # Page 195: socket **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/socket.md --- # socket --- title: socket latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/socket page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/socket.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/socket.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/socket/)[Input](/redpanda-connect/components/inputs/socket/) **Available in:** Self-Managed Connects to a (tcp/udp/unix) server and sends a continuous stream of data, dividing messages according to the specified codec. ```yml # Config fields, showing default values output: label: "" socket: network: "" # No default (required) address: /tmp/benthos.sock # No default (required) codec: lines ``` ## [](#fields)Fields ### [](#address)`address` The address to connect to. **Type**: `string` ```yaml # Examples: address: /tmp/benthos.sock # --- address: 127.0.0.1:6000 ``` ### [](#codec)`codec` The way in which the bytes of messages should be written out into the output data stream. It’s possible to write lines using a custom delimiter with the `delim:x` codec, where x is the character sequence custom delimiter. **Type**: `string` **Default**: `lines` | Option | Summary | | --- | --- | | all-bytes | Only applicable to file based outputs. Writes each message to a file in full, if the file already exists the old content is deleted. | | append | Append each message to the output stream without any delimiter or special encoding. | | lines | Append each message to the output stream followed by a line break. | | delim:x | Append each message to the output stream followed by a custom delimiter. | ```yaml # Examples: codec: lines # --- codec: delim: # --- codec: delim:foobar ``` ### [](#network)`network` A network type to connect as. **Type**: `string` **Options**: `unix`, `tcp`, `udp` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plaintext certificate to use for TLS authentication. Must be paired with the corresponding private key in the `key` field when using inline PEM data for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path to a file containing the certificate to use for TLS authentication. Must be paired with the corresponding private key file in the `key_file` field when using file-based configuration for mTLS client certificates. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` Private key for mTLS client certificate as inline PEM data. Must correspond to the client certificate specified in the `cert` field. Use this field together with `cert` when providing certificate data inline rather than through files. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` Path to private key file for mTLS client certificate in PEM format. Must correspond to the client certificate specified in the `cert_file` field. Use this field together with `cert_file` when loading certificate data from files. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The password to use for the private key (specified in the `key` or `key_file` fields), if it is password-protected. The PKCS#1 and PKCS#8 formats are supported. Supports environment variable interpolation for secure password management. The `pbeWithMD5AndDES-CBC` algorithm is obsolete and not supported for the PKCS#8 format. This algorithm does not authenticate the ciphertext, making it vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to enable TLS for secure connections. Set to `true` to enable TLS encryption. Required to be `true` for other TLS options (like `client_certs`, `root_cas`, etc.) to take effect. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` --- # Page 196: splunk_hec **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/splunk_hec.md --- # splunk\_hec --- title: splunk_hec latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/splunk_hec page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/splunk_hec.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/splunk_hec.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/splunk_hec/ "View the Cloud version of this component"), Self-Managed **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Publishes messages to a Splunk HTTP Endpoint Collector (HEC). Introduced in version 4.30.0. #### Common ```yml outputs: label: "" splunk_hec: url: "" # No default (required) token: "" # No default (required) gzip: false event_host: "" # No default (optional) event_source: "" # No default (optional) event_sourcetype: "" # No default (optional) event_index: "" # No default (optional) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" splunk_hec: url: "" # No default (required) token: "" # No default (required) gzip: false event_host: "" # No default (optional) event_source: "" # No default (optional) event_sourcetype: "" # No default (optional) event_index: "" # No default (optional) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#performance)Performance This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`. This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more [in this doc](../../../configuration/batching/). ## [](#fields)Fields ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#event_host)`event_host` Set the host value to assign to the event data. Overrides existing host field if present. **Type**: `string` ### [](#event_index)`event_index` Set the index value to assign to the event data. Overrides existing index field if present. **Type**: `string` ### [](#event_source)`event_source` Set the source value to assign to the event data. Overrides existing source field if present. **Type**: `string` ### [](#event_sourcetype)`event_sourcetype` Set the sourcetype value to assign to the event data. Overrides existing sourcetype field if present. **Type**: `string` ### [](#gzip)`gzip` Enable gzip compression **Type**: `bool` **Default**: `false` ### [](#max_in_flight)`max_in_flight` The maximum number of messages to have in flight at a given time. Increase this to improve throughput. **Type**: `int` **Default**: `64` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#token)`token` A bot token used for authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#url)`url` Full HTTP Endpoint Collector (HEC) URL. **Type**: `string` ```yaml # Examples: url: https://foobar.splunkcloud.com/services/collector/event ``` --- # Page 197: sql_insert **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/sql_insert.md --- # sql\_insert --- title: sql_insert latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/sql_insert page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/sql_insert.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/sql_insert.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-03-18" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/sql_insert/)[Processor](/redpanda-connect/components/processors/sql_insert/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/sql_insert/ "View the Cloud version of this component"), Self-Managed Inserts a row into an SQL database for each message. Introduced in version 3.59.0. #### Common ```yml outputs: label: "" sql_insert: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) columns: [] # No default (required) args_mapping: "" # No default (required) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" sql_insert: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) columns: [] # No default (required) args_mapping: "" # No default (required) prefix: "" # No default (optional) suffix: "" # No default (optional) options: [] # No default (optional) max_in_flight: 64 init_files: [] # No default (optional) init_statement: "" # No default (optional) conn_max_idle_time: "" # No default (optional) conn_max_life_time: "" # No default (optional) conn_max_idle: 2 conn_max_open: "" # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#examples)Examples ### [](#table-insert-mysql)Table Insert (MySQL) Here we insert rows into a database by populating the columns id, name and topic with values extracted from messages and metadata: ```yaml output: sql_insert: driver: mysql dsn: foouser:foopassword@tcp(localhost:3306)/foodb table: footable columns: [ id, name, topic ] args_mapping: | root = [ this.user.id, this.user.name, meta("kafka_topic"), ] ``` ## [](#dynamic-sql-operations)Dynamic SQL operations The `table` and `columns` fields are static strings that do not support Bloblang interpolation. For dynamic table names, dynamic column lists, DELETE operations, or any other SQL that `sql_insert` cannot express, use the [`sql_raw` output](../sql_raw/) instead. There is no dedicated `sql_delete` output. To delete rows, use `sql_raw` with a DELETE statement: ```yaml output: sql_raw: driver: postgres dsn: postgres://user:pass@localhost:5432/mydb?sslmode=disable query: "DELETE FROM my_table WHERE id = $1" args_mapping: root = [ this.id ] ``` To insert into a table determined at runtime, use `sql_raw` with `unsafe_dynamic_query: true`, which enables Bloblang interpolation in the `query` field. > ⚠️ **CAUTION** > > Interpolating user-supplied values into a query can introduce SQL injection risks. Always validate or sanitize the interpolated value beforehand. ```yaml output: sql_raw: driver: postgres dsn: postgres://user:pass@localhost:5432/mydb?sslmode=disable unsafe_dynamic_query: true query: 'INSERT INTO ${! this.table_name } (id, value) VALUES ($1, $2)' args_mapping: root = [ this.id, this.value ] ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` A [Bloblang mapping](../../../guides/bloblang/about/) which should evaluate to an array of values matching in size to the number of columns specified. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#columns)`columns[]` A list of columns to insert. **Type**: `array` ```yaml # Examples: columns: - foo - bar - baz ``` ### [](#conn_max_idle)`conn_max_idle` An optional maximum number of connections in the idle connection pool. If conn\_max\_open is greater than 0 but less than the new conn\_max\_idle, then the new conn\_max\_idle will be reduced to match the conn\_max\_open limit. If `value ⇐ 0`, no idle connections are retained. The default max idle connections is currently 2. This may change in a future release. **Type**: `int` **Default**: `2` ### [](#conn_max_idle_time)`conn_max_idle_time` An optional maximum amount of time a connection may be idle. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections idle time. **Type**: `string` ### [](#conn_max_life_time)`conn_max_life_time` An optional maximum amount of time a connection may be reused. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections age. **Type**: `string` ### [](#conn_max_open)`conn_max_open` An optional maximum number of open connections to the database. If conn\_max\_idle is greater than 0 and the new conn\_max\_open is less than conn\_max\_idle, then conn\_max\_idle will be reduced to match the new conn\_max\_open limit. If `value ⇐ 0`, then there is no limit on the number of open connections. The default is 0 (unlimited). **Type**: `int` ### [](#driver)`driver` A database [driver](#drivers) to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks` ### [](#dsn)`dsn` A Data Source Name to identify the target database. #### [](#drivers)Drivers The following is a list of supported drivers, their placeholder style, and their respective DSN formats: | Driver | Data Source Name Format | | --- | --- | | clickhouse | clickhouse://[username[:password]@][netloc][:port]/dbname[?param1=value1&…​¶mN=valueN] | | mysql | [username[:password]@][protocol[(address)]]/dbname[?param1=value1&…​¶mN=valueN] | | postgres and pgx | postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&…​] | | mssql | sqlserver://[user[:password]@][netloc][:port][?database=dbname¶m1=value1&…​] | | sqlite | file:/path/to/filename.db[?param&=value1&…​] | | oracle | oracle://[username[:password]@][netloc][:port]/service_name?server=server2&server=server3 | | snowflake | username[:password]@account_identifier/dbname/schemaname[?param1=value&…​¶mN=valueN] | | trino | http[s]://user[:pass]@host[:port][?parameters] | | gocosmos | AccountEndpoint=;AccountKey=[;TimeoutMs=][;Version=][;DefaultDb/Db=][;AutoId=][;InsecureSkipVerify=] | | spanner | projects/[PROJECT]/instances/[INSTANCE]/databases/[DATABASE] | | databricks | token:@:/ | Please note that the `postgres` and `pgx` drivers enforce SSL by default, you can override this with the parameter `sslmode=disable` if required. The `pgx` driver is an alternative to the standard `postgres` (pq) driver and comes with extra functionality such as support for array insertion. The `snowflake` driver supports multiple DSN formats. Please consult [the docs](https://pkg.go.dev/github.com/snowflakedb/gosnowflake#hdr-Connection_String) for more details. For [key pair authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication), the DSN has the following format: `@//?warehouse=&role=&authenticator=snowflake_jwt&privateKey=`, where the value for the `privateKey` parameter can be constructed from an unencrypted RSA private key file `rsa_key.p8` using `openssl enc -d -base64 -in rsa_key.p8 | basenc --base64url -w0` (you can use `gbasenc` instead of `basenc` on OSX if you install `coreutils` via Homebrew). If you have a password-encrypted private key, you can decrypt it using `openssl pkcs8 -in rsa_key_encrypted.p8 -out rsa_key.p8`. Also, make sure fields such as the username are URL-encoded. The [`gocosmos`](https://pkg.go.dev/github.com/microsoft/gocosmos) driver is still experimental, but it has support for [hierarchical partition keys](https://learn.microsoft.com/en-us/azure/cosmos-db/hierarchical-partition-keys) as well as [cross-partition queries](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-query-container#cross-partition-query). Please refer to the [SQL notes](https://github.com/microsoft/gocosmos/blob/main/SQL.md) for details. **Type**: `string` ```yaml # Examples: dsn: clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60 # --- dsn: foouser:foopassword@tcp(localhost:3306)/foodb # --- dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable # --- dsn: oracle://foouser:foopass@localhost:1521/service_name # --- dsn: token:dapi1234567890ab@dbc-a1b2345c-d6e7.cloud.databricks.com:443/sql/1.0/warehouses/abc123def456 ``` ### [](#init_files)`init_files[]` An optional list of file paths containing SQL statements to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Glob patterns are supported, including super globs (double star). Care should be taken to ensure that the statements are idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If a statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `array` ```yaml # Examples: init_files: - ./init/*.sql # --- init_files: - ./foo.sql - ./bar.sql ``` ### [](#init_statement)`init_statement` An optional SQL statement to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Care should be taken to ensure that the statement is idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If the statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `string` ```yaml # Examples: init_statement: |- CREATE TABLE IF NOT EXISTS some_table ( foo varchar(50) not null, bar integer, baz varchar(50), primary key (foo) ) WITHOUT ROWID; ``` ### [](#max_in_flight)`max_in_flight` The maximum number of inserts to run in parallel. **Type**: `int` **Default**: `64` ### [](#options)`options[]` A list of keyword options to add before the INTO clause of the query. **Type**: `array` ```yaml # Examples: options: - DELAYED - IGNORE ``` ### [](#prefix)`prefix` An optional prefix to prepend to the insert query (before INSERT). **Type**: `string` ### [](#suffix)`suffix` An optional suffix to append to the insert query. **Type**: `string` ```yaml # Examples: suffix: ON CONFLICT (name) DO NOTHING ``` ### [](#table)`table` The table to insert to. **Type**: `string` ```yaml # Examples: table: foo ``` --- # Page 198: sql_raw **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/sql_raw.md --- # sql\_raw --- title: sql_raw latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/sql_raw page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/sql_raw.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/sql_raw.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/sql_raw/)[Input](/redpanda-connect/components/inputs/sql_raw/)[Processor](/redpanda-connect/components/processors/sql_raw/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/sql_raw/ "View the Cloud version of this component"), Self-Managed Executes an arbitrary SQL query for each message. Introduced in version 3.65.0. #### Common ```yml outputs: label: "" sql_raw: driver: "" # No default (required) dsn: "" # No default (required) query: "" # No default (optional) args_mapping: "" # No default (optional) queries: [] # No default (optional) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" sql_raw: driver: "" # No default (required) dsn: "" # No default (required) query: "" # No default (optional) unsafe_dynamic_query: false args_mapping: "" # No default (optional) queries: [] # No default (optional) max_in_flight: 64 init_files: [] # No default (optional) init_statement: "" # No default (optional) conn_max_idle_time: "" # No default (optional) conn_max_life_time: "" # No default (optional) conn_max_idle: 2 conn_max_open: "" # No default (optional) batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` For some scenarios where you might use this output, see [Examples](#examples). ## [](#fields)Fields ### [](#args_mapping)`args_mapping` An optional [Bloblang mapping](../../../guides/bloblang/about/) that includes the same number of values in an array as the placeholder arguments in the [`query`](#query) field. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#conn_max_idle)`conn_max_idle` An optional maximum number of connections in the idle connection pool. If conn\_max\_open is greater than 0 but less than the new conn\_max\_idle, then the new conn\_max\_idle will be reduced to match the conn\_max\_open limit. If `value ⇐ 0`, no idle connections are retained. The default max idle connections is currently 2. This may change in a future release. **Type**: `int` **Default**: `2` ### [](#conn_max_idle_time)`conn_max_idle_time` An optional maximum amount of time a connection may be idle. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections idle time. **Type**: `string` ### [](#conn_max_life_time)`conn_max_life_time` An optional maximum amount of time a connection may be reused. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections age. **Type**: `string` ### [](#conn_max_open)`conn_max_open` An optional maximum number of open connections to the database. If conn\_max\_idle is greater than 0 and the new conn\_max\_open is less than conn\_max\_idle, then conn\_max\_idle will be reduced to match the new conn\_max\_open limit. If `value ⇐ 0`, then there is no limit on the number of open connections. The default is 0 (unlimited). **Type**: `int` ### [](#driver)`driver` A database [driver](#drivers) to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks` ### [](#dsn)`dsn` A Data Source Name to identify the target database. #### [](#drivers)Drivers The following is a list of supported drivers, their placeholder style, and their respective DSN formats: | Driver | Data Source Name Format | | --- | --- | | clickhouse | clickhouse://[username[:password]@][netloc][:port]/dbname[?param1=value1&…​¶mN=valueN] | | mysql | [username[:password]@][protocol[(address)]]/dbname[?param1=value1&…​¶mN=valueN] | | postgres and pgx | postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&…​] | | mssql | sqlserver://[user[:password]@][netloc][:port][?database=dbname¶m1=value1&…​] | | sqlite | file:/path/to/filename.db[?param&=value1&…​] | | oracle | oracle://[username[:password]@][netloc][:port]/service_name?server=server2&server=server3 | | snowflake | username[:password]@account_identifier/dbname/schemaname[?param1=value&…​¶mN=valueN] | | trino | http[s]://user[:pass]@host[:port][?parameters] | | gocosmos | AccountEndpoint=;AccountKey=[;TimeoutMs=][;Version=][;DefaultDb/Db=][;AutoId=][;InsecureSkipVerify=] | | spanner | projects/[PROJECT]/instances/[INSTANCE]/databases/[DATABASE] | | databricks | token:@:/ | Please note that the `postgres` and `pgx` drivers enforce SSL by default, you can override this with the parameter `sslmode=disable` if required. The `pgx` driver is an alternative to the standard `postgres` (pq) driver and comes with extra functionality such as support for array insertion. The `snowflake` driver supports multiple DSN formats. Please consult [the docs](https://pkg.go.dev/github.com/snowflakedb/gosnowflake#hdr-Connection_String) for more details. For [key pair authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication), the DSN has the following format: `@//?warehouse=&role=&authenticator=snowflake_jwt&privateKey=`, where the value for the `privateKey` parameter can be constructed from an unencrypted RSA private key file `rsa_key.p8` using `openssl enc -d -base64 -in rsa_key.p8 | basenc --base64url -w0` (you can use `gbasenc` instead of `basenc` on OSX if you install `coreutils` via Homebrew). If you have a password-encrypted private key, you can decrypt it using `openssl pkcs8 -in rsa_key_encrypted.p8 -out rsa_key.p8`. Also, make sure fields such as the username are URL-encoded. The [`gocosmos`](https://pkg.go.dev/github.com/microsoft/gocosmos) driver is still experimental, but it has support for [hierarchical partition keys](https://learn.microsoft.com/en-us/azure/cosmos-db/hierarchical-partition-keys) as well as [cross-partition queries](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-query-container#cross-partition-query). Please refer to the [SQL notes](https://github.com/microsoft/gocosmos/blob/main/SQL.md) for details. **Type**: `string` ```yaml # Examples: dsn: clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60 # --- dsn: foouser:foopassword@tcp(localhost:3306)/foodb # --- dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable # --- dsn: oracle://foouser:foopass@localhost:1521/service_name # --- dsn: token:dapi1234567890ab@dbc-a1b2345c-d6e7.cloud.databricks.com:443/sql/1.0/warehouses/abc123def456 ``` ### [](#init_files)`init_files[]` An optional list of file paths containing SQL statements to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Glob patterns are supported, including super globs (double star). Care should be taken to ensure that the statements are idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If a statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `array` ```yaml # Examples: init_files: - ./init/*.sql # --- init_files: - ./foo.sql - ./bar.sql ``` ### [](#init_statement)`init_statement` An optional SQL statement to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Care should be taken to ensure that the statement is idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If the statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `string` ```yaml # Examples: init_statement: |- CREATE TABLE IF NOT EXISTS some_table ( foo varchar(50) not null, bar integer, baz varchar(50), primary key (foo) ) WITHOUT ROWID; ``` ### [](#max_in_flight)`max_in_flight` The maximum number of database statements to execute in parallel. **Type**: `int` **Default**: `64` ### [](#queries)`queries[]` A list of database statements to run in addition to your main [`query`](#query). If you specify multiple queries, they are executed within a single transaction. For more information, see [Examples](#examples). **Type**: `object` ### [](#queries-args_mapping)`queries[].args_mapping` An optional [Bloblang mapping](../../../guides/bloblang/about/) which should evaluate to an array of values matching in size to the number of placeholder arguments in the field `query`. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#queries-query)`queries[].query` The query to execute. The style of placeholder to use depends on the driver, some drivers require question marks (`?`) whereas others expect incrementing dollar signs (`$1`, `$2`, and so on) or colons (`:1`, `:2` and so on). The style to use is outlined in this table: | Driver | Placeholder Style | |---|---| | `clickhouse` | Dollar sign | | `mysql` | Question mark | | `postgres` | Dollar sign | | `pgx` | Dollar sign | | `mssql` | Question mark | | `sqlite` | Question mark | | `oracle` | Colon | | `snowflake` | Question mark | | `trino` | Question mark | | `gocosmos` | Colon | **Type**: `string` ### [](#queries-when)`queries[].when` An optional [Bloblang mapping](../../../guides/bloblang/about/) that, when set, is evaluated for each message to determine whether to execute this query. The mapping should return a boolean value. The first query in the list whose `when` condition evaluates to `true` (or that has no `when` condition) is executed. This enables conditional query routing based on message content or metadata without requiring `unsafe_dynamic_query`. **Type**: `string` ```yaml # Examples: when: root = meta("kafka_tombstone_message") == "true" # --- when: root = this.operation == "delete" ``` ### [](#query)`query` The query to execute. You must include the correct placeholders for the specified database driver. Some drivers use question marks (`?`), whereas others expect incrementing dollar signs (`$1`, `$2`, and so on) or colons (`:1`, `:2`, and so on). | Driver | Placeholder Style | | --- | --- | | clickhouse | Dollar sign ($) | | gocosmos | Colon (:) | | mysql | Question mark (?) | | mssql | Question mark (?) | | oracle | Colon (:) | | postgres | Dollar sign ($) | | snowflake | Question mark (?) | | spanner | Question mark (?) | | sqlite | Question mark (?) | | trino | Question mark (?) | **Type**: `string` ```yaml # Examples: query: INSERT INTO footable (foo, bar, baz) VALUES (?, ?, ?); ``` ### [](#unsafe_dynamic_query)`unsafe_dynamic_query` Whether to enable [interpolation functions](../../../configuration/interpolation/#bloblang-queries) in the query. Great care should be made to ensure your queries are defended against injection attacks. **Type**: `bool` **Default**: `false` ## [](#examples)Examples ### [](#table-insert-mysql)Table Insert (MySQL) Here we insert rows into a database by populating the columns id, name and topic with values extracted from messages and metadata: ```yaml output: sql_raw: driver: mysql dsn: foouser:foopassword@tcp(localhost:3306)/foodb query: "INSERT INTO footable (id, name, topic) VALUES (?, ?, ?);" args_mapping: | root = [ this.user.id, this.user.name, meta("kafka_topic"), ] ``` ### [](#dynamically-creating-tables-postgresql)Dynamically Creating Tables (PostgreSQL) Here we dynamically create output tables transactionally with inserting a record into the newly created table. ```yaml output: processors: - mapping: | root = this # Prevent SQL injection when using unsafe_dynamic_query meta table_name = "\"" + metadata("table_name").replace_all("\"", "\"\"") + "\"" sql_raw: driver: postgres dsn: postgres://localhost/postgres unsafe_dynamic_query: true queries: - query: | CREATE TABLE IF NOT EXISTS ${!metadata("table_name")} (id varchar primary key, document jsonb); - query: | INSERT INTO ${!metadata("table_name")} (id, document) VALUES ($1, $2) ON CONFLICT (id) DO UPDATE SET document = EXCLUDED.document; args_mapping: | root = [ this.id, this.document.string() ] ``` ### [](#conditional-cdc-queries-postgresql)Conditional CDC Queries (PostgreSQL) Route messages to different SQL operations based on message metadata. Tombstone messages trigger a DELETE, while all other messages perform an upsert. All operations within a batch execute in a single transaction, ordered by Kafka partition. ```yaml output: sql_raw: driver: postgres dsn: postgres://localhost/postgres max_in_flight: 8 batching: count: 100 period: 100ms queries: - when: 'root = meta("kafka_tombstone_message") == "true"' query: 'DELETE FROM users WHERE id = $1' args_mapping: 'root = [this.id]' - query: | INSERT INTO users (id, name, updated_at) VALUES ($1, $2, $3) ON CONFLICT (id) DO UPDATE SET name = EXCLUDED.name, updated_at = EXCLUDED.updated_at args_mapping: 'root = [this.id, this.name, this.updated_at]' ``` --- # Page 199: sql **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/sql.md --- # sql --- title: sql latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/sql page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/sql.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/sql.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/sql/)[Cache](/redpanda-connect/components/caches/sql/)[Processor](/redpanda-connect/components/processors/sql/) **Available in:** Self-Managed > ⚠️ **WARNING: Deprecated** > > Deprecated > > This component is deprecated and will be removed in the next major version release. Please consider moving onto [alternative components](#alternatives). Executes an arbitrary SQL query for each message. Introduced in version 3.65.0. #### Common ```yml outputs: label: "" sql: driver: "" # No default (required) data_source_name: "" # No default (required) query: "" # No default (required) args_mapping: "" # No default (optional) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` #### Advanced ```yml outputs: label: "" sql: driver: "" # No default (required) data_source_name: "" # No default (required) query: "" # No default (required) args_mapping: "" # No default (optional) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` ## [](#alternatives)Alternatives For basic inserts use the [`sql_insert`](../sql_insert/) output. For more complex queries use the [`sql_raw`](../sql_raw/) output. ## [](#fields)Fields ### [](#args_mapping)`args_mapping` An optional [Bloblang mapping](../../../guides/bloblang/about/) which should evaluate to an array of values matching in size to the number of placeholder arguments in the field `query`. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#batching)`batching` Allows you to configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yaml # Examples: batching: byte_size: 5000 count: 0 period: 1s # --- batching: count: 10 period: 1s # --- batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-byte_size)`batching.byte_size` An amount of bytes at which the batch should be flushed. If `0` disables size based batching. **Type**: `int` **Default**: `0` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "end_of_transaction" ``` ### [](#batching-count)`batching.count` A number of messages at which the batch should be flushed. If `0` disables count based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` A period in which an incomplete batch should be flushed regardless of its size. **Type**: `string` **Default**: `""` ```yaml # Examples: period: 1s # --- period: 1m # --- period: 500ms ``` ### [](#batching-processors)`batching.processors[]` A list of [processors](../../processors/about/) to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op. **Type**: `processor` ```yaml # Examples: processors: - archive: format: concatenate # --- processors: - archive: format: lines # --- processors: - archive: format: json_array ``` ### [](#data_source_name)`data_source_name` Data source name. **Type**: `string` ### [](#driver)`driver` A database [driver](#drivers) to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks` ### [](#max_in_flight)`max_in_flight` The maximum number of inserts to run in parallel. **Type**: `int` **Default**: `64` ### [](#query)`query` The query to execute. The style of placeholder to use depends on the driver, some drivers require question marks (`?`) whereas others expect incrementing dollar signs (`$1`, `$2`, and so on) or colons (`:1`, `:2` and so on). The style to use is outlined in this table: | Driver | Placeholder Style | | --- | --- | | clickhouse | Dollar sign ($) | | gocosmos | Colon (:) | | mysql | Question mark (?) | | mssql | Question mark (?) | | oracle | Colon (:) | | postgres | Dollar sign ($) | | snowflake | Question mark (?) | | spanner | Question mark (?) | | sqlite | Question mark (?) | | trino | Question mark (?) | **Type**: `string` ```yaml # Examples: query: INSERT INTO footable (foo, bar, baz) VALUES (?, ?, ?); ``` --- # Page 200: stdout **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/stdout.md --- # stdout --- title: stdout latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/stdout page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/stdout.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/stdout.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** Self-Managed Prints messages to stdout as a continuous stream of data. #### Common ```yml outputs: label: "" stdout: codec: lines ``` #### Advanced ```yml outputs: label: "" stdout: codec: lines ``` ## [](#fields)Fields ### [](#codec)`codec` The way in which the bytes of messages should be written out into the output data stream. It’s possible to write lines using a custom delimiter with the `delim:x` codec, where x is the character sequence custom delimiter. Requires version 3.46.0 or later. **Type**: `string` **Default**: `lines` | Option | Summary | | --- | --- | | all-bytes | Only applicable to file based outputs. Writes each message to a file in full, if the file already exists the old content is deleted. | | append | Append each message to the output stream without any delimiter or special encoding. | | lines | Append each message to the output stream followed by a line break. | | delim:x | Append each message to the output stream followed by a custom delimiter. | ```yaml # Examples: codec: lines # --- codec: delim: # --- codec: delim:foobar ``` --- # Page 201: subprocess **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/subprocess.md --- # subprocess --- title: subprocess latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/subprocess page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/subprocess.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/subprocess.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/subprocess/)[Input](/redpanda-connect/components/inputs/subprocess/)[Processor](/redpanda-connect/components/processors/subprocess/) **Available in:** Self-Managed Executes a command, runs it as a subprocess, and writes messages to it over stdin. ```yml # Config fields, showing default values output: label: "" subprocess: name: "" # No default (required) args: [] codec: lines ``` Messages are written according to a specified codec. The process is expected to terminate gracefully when stdin is closed. If the subprocess exits unexpectedly then Redpanda Connect will log anything printed to stderr and will log the exit code, and will attempt to execute the command again until success. The execution environment of the subprocess is the same as the Redpanda Connect instance, including environment variables and the current working directory. ## [](#fields)Fields ### [](#args)`args[]` A list of arguments to provide the command. **Type**: `array` **Default**: `[]` ### [](#codec)`codec` The way in which messages should be written to the subprocess. **Type**: `string` **Default**: `lines` **Options**: `lines` ### [](#name)`name` The command to execute as a subprocess. **Type**: `string` --- # Page 202: switch **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/switch.md --- # switch --- title: switch latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/switch page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/switch.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/switch.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/switch/)[Processor](/redpanda-connect/components/processors/switch/)[Scanner](/redpanda-connect/components/scanners/switch/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/switch/ "View the Cloud version of this component"), Self-Managed The switch output type allows you to route messages to different outputs based on their contents. #### Common ```yml outputs: label: "" switch: retry_until_success: false cases: [] # No default (required) ``` #### Advanced ```yml outputs: label: "" switch: retry_until_success: false strict_mode: false cases: [] # No default (required) ``` Messages that do not pass the check of a single output case are effectively dropped. In order to prevent this outcome set the field [`strict_mode`](#strict_mode) to `true`, in which case messages that do not pass at least one case are considered failed and will be nacked and/or reprocessed depending on your input. ## [](#examples)Examples ### [](#basic-multiplexing)Basic Multiplexing The most common use for a switch output is to multiplex messages across a range of output destinations. The following config checks the contents of the field `type` of messages and sends `foo` type messages to an `amqp_1` output, `bar` type messages to a `gcp_pubsub` output, and everything else to a `redis_streams` output. Outputs can have their own processors associated with them, and in this example the `redis_streams` output has a processor that enforces the presence of a type field before sending it. ```yaml output: switch: cases: - check: this.type == "foo" output: amqp_1: urls: [ amqps://guest:guest@localhost:5672/ ] target_address: queue:/the_foos - check: this.type == "bar" output: gcp_pubsub: project: dealing_with_mike topic: mikes_bars - output: redis_streams: url: tcp://localhost:6379 stream: everything_else processors: - mapping: | root = this root.type = this.type | "unknown" ``` ### [](#control-flow)Control Flow The `continue` field allows messages that have passed a case to be tested against the next one also. This can be useful when combining non-mutually-exclusive case checks. In the following example a message that passes both the check of the first case as well as the second will be routed to both. ```yaml output: switch: cases: - check: 'this.user.interests.contains("walks").catch(false)' output: amqp_1: urls: [ amqps://guest:guest@localhost:5672/ ] target_address: queue:/people_what_think_good continue: true - check: 'this.user.dislikes.contains("videogames").catch(false)' output: gcp_pubsub: project: people topic: that_i_dont_want_to_hang_with ``` ## [](#fields)Fields ### [](#cases)`cases[]` A list of switch cases, outlining outputs that can be routed to. **Type**: `object` ```yaml # Examples: cases: - check: this.urls.contains("http://benthos.dev") continue: true output: cache: key: ${!json("id")} target: foo - output: s3: bucket: bar path: ${!json("id")} ``` ### [](#cases-check)`cases[].check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should be routed to the case output. If left empty the case always passes. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "foo" # --- check: this.contents.urls.contains("https://benthos.dev/") ``` ### [](#cases-continue)`cases[].continue` Indicates whether, if this case passes for a message, the next case should also be tested. **Type**: `bool` **Default**: `false` ### [](#cases-output)`cases[].output` An [output](../about/) for messages that pass the check to be routed to. **Type**: `output` ### [](#retry_until_success)`retry_until_success` If a selected output fails to send a message this field determines whether it is reattempted indefinitely. If set to false the error is instead propagated back to the input level. If a message can be routed to >1 outputs it is usually best to set this to true in order to avoid duplicate messages being routed to an output. **Type**: `bool` **Default**: `false` ### [](#strict_mode)`strict_mode` This field determines whether an error should be reported if no condition is met. If set to true, an error is propagated back to the input level. The default behavior is false, which will drop the message. **Type**: `bool` **Default**: `false` --- # Page 203: sync_response **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/sync_response.md --- # sync\_response --- title: sync_response latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/sync_response page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/sync_response.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/sync_response.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/sync_response/)[Processor](/redpanda-connect/components/processors/sync_response/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/sync_response/ "View the Cloud version of this component"), Self-Managed Returns the final message payload back to the input origin of the message, where it is dealt with according to that specific input type. ```yml # Config fields, showing default values output: label: "" sync_response: {} ``` For most inputs this mechanism is ignored entirely, in which case the sync response is dropped without penalty. It is therefore safe to use this output even when combining input types that might not have support for sync responses. An example of an input able to utilize this is the `http_server`. It is safe to combine this output with others using broker types. For example, with the `http_server` input we could send the payload to a Kafka topic and also send a modified payload back with: ```yaml input: http_server: path: /post output: broker: pattern: fan_out outputs: - kafka: addresses: [ TODO:9092 ] topic: foo_topic - sync_response: {} processors: - mapping: 'root = content().uppercase()' ``` Using the above example and posting the message 'hello world' to the endpoint `/post` Redpanda Connect would send it unchanged to the topic `foo_topic` and also respond with 'HELLO WORLD'. For more information please read [synchronous responses](../../../guides/sync_responses/). --- # Page 204: timeplus **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/timeplus.md --- # timeplus --- title: timeplus latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/timeplus page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/timeplus.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/timeplus.adoc page-git-created-date: "2024-11-05" page-git-modified-date: "2025-06-25" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/timeplus/)[Input](/redpanda-connect/components/inputs/timeplus/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/outputs/timeplus/ "View the Cloud version of this component"), Self-Managed Sends messages to a data stream on [Timeplus Enterprise (Cloud or Self-Hosted)](https://docs.timeplus.com/) using the [Ingest API](https://docs.timeplus.com/ingest-api), or directly to the `timeplusd` component in Timeplus Enterprise. #### Common ```yml # Common configuration fields, showing default values output: label: "" timeplus: target: timeplus url: https://us-west-2.timeplus.cloud workspace: "" # No default (optional) stream: "" # No default (required) apikey: "" # No default (optional) username: "" # No default (optional) password: "" # No default (optional) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" ``` #### Advanced ```yml # All configuration fields, showing default values output: label: "" timeplus: target: timeplus url: https://us-west-2.timeplus.cloud workspace: "" # No default (optional) stream: "" # No default (required) apikey: "" # No default (optional) username: "" # No default (optional) password: "" # No default (optional) max_in_flight: 64 batching: count: 0 byte_size: 0 period: "" check: "" processors: [] # No default (optional) ``` This output only accepts structured messages. All messages must: - Contain the same keys. - Use a structure that matches the schema of the destination data stream. If your upstream data source or pipeline returns unstructured messages, such as strings, you can configure an output processor to transform the messages. See the [Unstructured messages](#unstructured-messages) section for examples. ## [](#examples)Examples #### Timeplus Enterprise (Cloud) You must [generate an API key](https://docs.timeplus.com/apikey) using the web console of Timeplus Enterprise (Cloud). ```yaml output: timeplus: workspace: stream: apikey: ``` Replace the following placeholders with your own values: - ``: The ID of the workspace you want to send messages to. - ``: The name of the destination data stream. - ``: The API key for the Ingest API. #### Timeplus Enterprise (Self-Hosted) You must specify the username, password, and URL of the application server. ```yaml output: timeplus: url: http://localhost:8000 workspace: stream: username: password: ``` Replace the following placeholders with your own values: - ``: The ID of the workspace you want to send messages to. - ``: The name of the destination data stream. - ``: The username for the Timeplus application server. - ``: The password for the Timeplus application server. #### timeplusd You must specify the HTTP port for `timeplusd`. ```yaml output: timeplus: url: http://localhost:3218 stream: username: password: ``` Replace the following placeholders with your own values: - ``: The name of the destination data stream. - ``: The username for the Timeplus application server. - ``: The password for the Timeplus application server. ### [](#unstructured-messages)Unstructured messages If your upstream data source or pipeline returns unstructured messages, such as strings, you can configure an output processor to transform them into structured messages and then pass them to the output. In the following example, the `mapping` processor creates a field called `raw`, and uses the functions `content().string()` to store the original string content into it, thereby creating structured messages. If you use this example, you must also add the `raw` field name to the destination data stream, so that your message structure matches the schema of your destination data stream. ```yaml output: timeplus: workspace: stream: apikey: processors: - mapping: | root = {} root.raw = content().string() ``` ## [](#fields)Fields ### [](#target)`target` The destination platform. For Timeplus Enterprise (Cloud or Self-Hosted), enter `timeplus`, or `timeplusd` for the `timeplusd` component. **Type**: `string` **Default**: `timeplus` **Options**: `timeplus`, `timeplusd` ### [](#url)`url` The URL of your Timeplus instance, which should always include the schema and host. **Type**: `string` **Default**: `[https://us-west-2.timeplus.cloud](https://us-west-2.timeplus.cloud)` ```yml # Examples url: http://localhost:8000 url: http://127.0.0.1:3218 ``` ### [](#workspace)`workspace` The ID of the workspace you want to send messages to. This field is required if the `target` field is set to `timeplus`. **Type**: `string` ### [](#stream)`stream` The name of the destination data stream. Make sure the schema of the data stream matches this output. **Type**: `string` ### [](#apikey)`apikey` The API key for the Ingest API. You need to generate this in the web console of Timeplus Enterprise (Cloud). This field is required if you are sending messages to Timeplus Enterprise (Cloud). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#username)`username` The username for the Timeplus application server. This field is required if you are sending messages to Timeplus Enterprise (Self-Hosted) or `timeplusd`. **Type**: `string` ### [](#password)`password` The password for the Timeplus application server. This field is required if you are sending messages to Timeplus Enterprise (Self-Hosted) or `timeplusd`. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#max_in_flight)`max_in_flight` The maximum number of message batches to have in flight at a given time. Increase this number to improve throughput. **Type**: `int` **Default**: `64` ### [](#batching)`batching` Configure a [batching policy](../../../configuration/batching/). **Type**: `object` ```yml # Examples batching: byte_size: 5000 count: 0 period: 1s batching: count: 10 period: 1s batching: check: this.contains("END BATCH") count: 0 period: 1m ``` ### [](#batching-count)`batching.count` The number of messages after which the batch is flushed. Set to `0` to disable count-based batching. **Type**: `int` **Default**: `0` ### [](#batching-byte_size)`batching.byte_size` The amount of bytes at which the batch is flushed. Set to `0` to disable size-based batching. **Type**: `int` **Default**: `0` ### [](#batching-period)`batching.period` The period of time after which an incomplete batch is flushed regardless of its size. **Type**: `string` **Default**: `""` ```yml # Examples period: 1s period: 1m period: 500ms ``` ### [](#batching-check)`batching.check` A [Bloblang query](../../../guides/bloblang/about/) that returns a boolean value indicating whether a message should end a batch. **Type**: `string` **Default**: `""` ```yml # Examples check: this.type == "end_of_transaction" ``` ### [](#batching-processors)`batching.processors` For aggregating and archiving message batches, you can add a list of [processors](../../processors/about/) to apply to a batch as it is flushed. All resulting messages are flushed as a single batch even when you configure processors to split the batch into smaller batches. **Type**: `array` ```yml # Examples processors: - archive: format: concatenate processors: - archive: format: lines processors: - archive: format: json_array ``` --- # Page 205: websocket **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/websocket.md --- # websocket --- title: websocket latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/websocket page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/websocket.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/websocket.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/websocket/)[Input](/redpanda-connect/components/inputs/websocket/) **Available in:** Self-Managed Sends messages to an HTTP server via a websocket connection. #### Common ```yml outputs: label: "" websocket: url: "" # No default (required) ``` #### Advanced ```yml outputs: label: "" websocket: url: "" # No default (required) proxy_url: "" # No default (optional) tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} ``` ## [](#fields)Fields ### [](#basic_auth)`basic_auth` Allows you to specify basic authentication. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#jwt)`jwt` Beta Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWT tokens provide secure, stateless authentication between services. **Type**: `object` ### [](#jwt-claims)`jwt.claims` A value used to identify the claims that issued the JWT. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` The cryptographic algorithm used to sign the JWT token. Supported algorithms include RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#oauth)`oauth` Configure OAuth version 1.0 authentication for secure API access. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` A value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` A secret provided in order to establish ownership of a given access token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` A value used to identify the client to the service provider. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` A secret used to establish ownership of the consumer key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#proxy_url)`proxy_url` An optional HTTP proxy URL. **Type**: `string` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL to connect to. **Type**: `string` --- # Page 206: zmq4 **URL**: https://docs.redpanda.com/redpanda-connect/components/outputs/zmq4.md --- # zmq4 --- title: zmq4 latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: outputs/zmq4 page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: outputs/zmq4.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/outputs/zmq4.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-08-15" --- **Type:** Output ▼ [Output](/redpanda-connect/components/outputs/zmq4/)[Input](/redpanda-connect/components/inputs/zmq4/) **Available in:** Self-Managed Writes messages to a ZeroMQ socket. #### Common ```yml # Common config fields, showing default values output: label: "" zmq4: urls: [] # No default (required) bind: true socket_type: "" # No default (required) ``` #### Advanced ```yml # All config fields, showing default values output: label: "" zmq4: urls: [] # No default (required) bind: true socket_type: "" # No default (required) high_water_mark: 0 poll_timeout: 5s ``` By default Redpanda Connect does not build with components that require linking to external libraries. If you wish to build Redpanda Connect locally with this component then set the build tag `x_benthos_extra`: ```bash # With go go install -tags "x_benthos_extra" github.com/redpanda-data/benthos/v4/cmd/benthos@latest # Using make make TAGS=x_benthos_extra ``` There is a specific docker tag postfix `-cgo` for C builds containing this component. ## [](#fields)Fields ### [](#urls)`urls` A list of URLs to connect to. If an item of the list contains commas it will be expanded into multiple URLs. **Type**: `array` ```yml # Examples urls: - tcp://localhost:5556 ``` ### [](#bind)`bind` Whether to bind to the specified URLs (otherwise they are connected to). **Type**: `bool` **Default**: `true` ### [](#socket_type)`socket_type` The socket type to connect as. **Type**: `string` Options: `PUSH` , `PUB` . ### [](#high_water_mark)`high_water_mark` The message high water mark to use. **Type**: `int` **Default**: `0` ### [](#poll_timeout)`poll_timeout` The poll timeout to use. **Type**: `string` **Default**: `"5s"` --- # Page 207: Processors **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/about.md --- # Processors --- title: Processors latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/about page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/about.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-03-31" --- Redpanda Connect processors are functions applied to messages passing through a pipeline. The function signature allows a processor to mutate or drop messages depending on the content of the message. There are many types on offer but the most powerful are the [`mapping`](../mapping/) and [`mutation`](../mutation/) processors. Processors are set via config, and depending on where in the config they are placed they will be run either immediately after a specific input (set in the input section), on all messages (set in the pipeline section) or before a specific output (set in the output section). Most processors apply to all messages and can be placed in the pipeline section: ```yaml pipeline: threads: 1 processors: - label: my_cool_mapping mapping: | root.message = this root.meta.link_count = this.links.length() ``` The `threads` field in the pipeline section determines how many parallel processing threads are created. You can read more about parallel processing in the [pipeline guide](../../../configuration/processing_pipelines/). ## [](#labels)Labels Processors have an optional field `label` that can uniquely identify them in observability data such as metrics and logs. This can be useful when running configs with multiple nested processors, otherwise their metrics labels will be generated based on their composition. For more information check out the [metrics documentation](../../metrics/about/). ## [](#error-handling)Error handling Some processors have conditions whereby they might fail. Rather than throw these messages into the abyss Redpanda Connect still attempts to send these messages onwards, and has mechanisms for filtering, recovering or dead-letter queuing messages that have failed which can be read about [here](../../../configuration/error_handling/). ### [](#error-logs)Error logs Errors that occur during processing can be roughly separated into two groups; those that are unexpected intermittent errors such as connectivity problems, and those that are logical errors such as bad input data or unmatched schemas. All processing errors result in the messages being flagged as failed, [error metrics](../../metrics/about/) increasing for the given errored processor, and debug level logs being emitted that describe the error. Only errors that are known to be intermittent are also logged at the error level. The reason for this behavior is to prevent noisy logging in cases where logical errors are expected and will likely be [handled in config](../../../configuration/error_handling/). However, this can also sometimes make it easy to miss logical errors in your configs when they lack error handling. If you suspect you are experiencing processing errors and do not wish to add error handling yet then a quick and easy way to expose those errors is to enable debug level logs with the cli flag `--log.level=debug` or by setting the level in config: ```yaml logger: level: DEBUG ``` ## [](#using-processors-as-outputs)Using processors as outputs It might be the case that a processor that results in a side effect, such as the [`sql_insert`](../sql_insert/) or [`redis`](../redis/) processors, is the only side effect of a pipeline, and therefore could be considered the output. In such cases it’s possible to place these processors within a [`reject` output](../../outputs/reject/) so that they behave the same as regular outputs, where success results in dropping the message with an acknowledgement and failure results in a nack (or retry): ```yaml output: reject: 'failed to send data: ${! error() }' processors: - try: - redis: url: tcp://localhost:6379 command: sadd args_mapping: 'root = [ this.key, this.value ]' - mapping: root = deleted() ``` The way this works is that if your processor with the side effect (`redis` in this case) succeeds then the final `mapping` processor deletes the message which results in an acknowledgement. If the processor fails then the `try` block exits early without executing the `mapping` processor and instead the message is routed to the `reject` output, which nacks the message with an error message containing the error obtained from the `redis` processor. ## [](#categories)Categories ### Parsing Processors that specialize in translating messages from one format to another. [**ZIP**](/redpanda-connect/components/processors/archive/) [**avro**](/redpanda-connect/components/processors/avro/) [**bloblang**](/redpanda-connect/components/processors/bloblang/) [**compress**](/redpanda-connect/components/processors/compress/) [**decompress**](/redpanda-connect/components/processors/decompress/) [**grok**](/redpanda-connect/components/processors/grok/) [**mapping**](/redpanda-connect/components/processors/mapping/) [**msgpack**](/redpanda-connect/components/processors/msgpack/) [**mutation**](/redpanda-connect/components/processors/mutation/) [**parquet\_decode**](/redpanda-connect/components/processors/parquet_decode/) [**parquet\_encode**](/redpanda-connect/components/processors/parquet_encode/) [**parse\_log**](/redpanda-connect/components/processors/parse_log/) [**protobuf**](/redpanda-connect/components/processors/protobuf/) [**schema\_registry\_decode**](/redpanda-connect/components/processors/schema_registry_decode/) [**schema\_registry\_encode**](/redpanda-connect/components/processors/schema_registry_encode/) [**string\_split**](/redpanda-connect/components/processors/string_split/) [**ZIP**](/redpanda-connect/components/processors/unarchive/) [**xml**](/redpanda-connect/components/processors/xml/) ### Utility Processors that provide general utility or do not fit in another category. [**ZIP**](/redpanda-connect/components/processors/archive/) [**benchmark**](/redpanda-connect/components/processors/benchmark/) [**bounds\_check**](/redpanda-connect/components/processors/bounds_check/) [**cached**](/redpanda-connect/components/processors/cached/) [**crash**](/redpanda-connect/components/processors/crash/) [**dedupe**](/redpanda-connect/components/processors/dedupe/) [**log**](/redpanda-connect/components/processors/log/) [**metric**](/redpanda-connect/components/processors/metric/) [**rate\_limit**](/redpanda-connect/components/processors/rate_limit/) [**redpanda\_data\_transform**](/redpanda-connect/components/processors/redpanda_data_transform/) [**resource**](/redpanda-connect/components/processors/resource/) [**select\_parts**](/redpanda-connect/components/processors/select_parts/) [**sleep**](/redpanda-connect/components/processors/sleep/) [**split**](/redpanda-connect/components/processors/split/) [**sync\_response**](/redpanda-connect/components/processors/sync_response/) [**ZIP**](/redpanda-connect/components/processors/unarchive/) [**wasm**](/redpanda-connect/components/processors/wasm/) ### Mapping Processors that specialize in restructuring messages. [**awk**](/redpanda-connect/components/processors/awk/) [**bloblang**](/redpanda-connect/components/processors/bloblang/) [**javascript**](/redpanda-connect/components/processors/javascript/) [**jmespath**](/redpanda-connect/components/processors/jmespath/) [**jq**](/redpanda-connect/components/processors/jq/) [**mapping**](/redpanda-connect/components/processors/mapping/) [**JSON Schema**](/redpanda-connect/components/processors/json_schema/) [**mutation**](/redpanda-connect/components/processors/mutation/) ### AI [**Amazon**](/redpanda-connect/components/processors/aws_bedrock_chat/) [**Amazon**](/redpanda-connect/components/processors/aws_bedrock_embeddings/) [**cohere\_chat**](/redpanda-connect/components/processors/cohere_chat/) [**cohere\_embeddings**](/redpanda-connect/components/processors/cohere_embeddings/) [**cohere\_rerank**](/redpanda-connect/components/processors/cohere_rerank/) [**GCP Vertex AI**](/redpanda-connect/components/processors/gcp_vertex_ai_chat/) [**Google Cloud**](/redpanda-connect/components/processors/gcp_vertex_ai_embeddings/) [**google\_drive\_download**](/redpanda-connect/components/processors/google_drive_download/) [**google\_drive\_list\_labels**](/redpanda-connect/components/processors/google_drive_list_labels/) [**google\_drive\_search**](/redpanda-connect/components/processors/google_drive_search/) [**ollama\_chat**](/redpanda-connect/components/processors/ollama_chat/) [**ollama\_embeddings**](/redpanda-connect/components/processors/ollama_embeddings/) [**ollama\_moderation**](/redpanda-connect/components/processors/ollama_moderation/) [**openai\_chat\_completion**](/redpanda-connect/components/processors/openai_chat_completion/) [**openai\_embeddings**](/redpanda-connect/components/processors/openai_embeddings/) [**openai\_image\_generation**](/redpanda-connect/components/processors/openai_image_generation/) [**openai\_speech**](/redpanda-connect/components/processors/openai_speech/) [**openai\_transcription**](/redpanda-connect/components/processors/openai_transcription/) [**openai\_translation**](/redpanda-connect/components/processors/openai_translation/) [**qdrant**](/redpanda-connect/components/processors/qdrant/) [**text\_chunker**](/redpanda-connect/components/processors/text_chunker/) ### Integration Processors that interact with external services. [**Amazon**](/redpanda-connect/components/processors/aws_dynamodb_partiql/) [**AWS Lambda**](/redpanda-connect/components/processors/aws_lambda/) [**cache**](/redpanda-connect/components/processors/cache/) [**command**](/redpanda-connect/components/processors/command/) [**couchbase**](/redpanda-connect/components/processors/couchbase/) [**GCP BigQuery**](/redpanda-connect/components/processors/gcp_bigquery_select/) [**http**](/redpanda-connect/components/processors/http/) [**redis**](/redpanda-connect/components/processors/redis/) [**Redis Script**](/redpanda-connect/components/processors/redis_script/) [**schema\_registry\_decode**](/redpanda-connect/components/processors/schema_registry_decode/) [**schema\_registry\_encode**](/redpanda-connect/components/processors/schema_registry_encode/) [**SQL**](/redpanda-connect/components/processors/sql_insert/) [**SQL**](/redpanda-connect/components/processors/sql_raw/) [**SQL**](/redpanda-connect/components/processors/sql_select/) [**subprocess**](/redpanda-connect/components/processors/subprocess/) ### Azure [**Microsoft Azure**](/redpanda-connect/components/processors/azure_cosmosdb/) ### Composition Higher level processors that compose other processors and modify their behavior. [**branch**](/redpanda-connect/components/processors/branch/) [**catch**](/redpanda-connect/components/processors/catch/) [**for\_each**](/redpanda-connect/components/processors/for_each/) [**group\_by**](/redpanda-connect/components/processors/group_by/) [**group\_by\_value**](/redpanda-connect/components/processors/group_by_value/) [**insert\_part**](/redpanda-connect/components/processors/insert_part/) [**parallel**](/redpanda-connect/components/processors/parallel/) [**processors**](/redpanda-connect/components/processors/processors/) [**retry**](/redpanda-connect/components/processors/retry/) [**switch**](/redpanda-connect/components/processors/switch/) [**try**](/redpanda-connect/components/processors/try/) [**while**](/redpanda-connect/components/processors/while/) [**workflow**](/redpanda-connect/components/processors/workflow/) ### [**Foreign Function Interface**](/redpanda-connect/components/processors/ffi/) ### Services [**jira**](/redpanda-connect/components/processors/jira/) [**MongoDB**](/redpanda-connect/components/processors/mongodb/) [**NATS KV**](/redpanda-connect/components/processors/nats_kv/) [**NATS Request Reply**](/redpanda-connect/components/processors/nats_request_reply/) [**Salesforce**](/redpanda-connect/components/processors/salesforce/) ## [](#batching-and-multiple-part-messages)Batching and multiple-part messages All Redpanda Connect processors support multiple-part messages, which are synonymous with batches. This enables [windowed processing](../../../configuration/windowed_processing/) capabilities. Many processors are able to perform their behaviors on specific parts of a message batch, or on all parts, and have a field `parts` for specifying an array of part indexes they should apply to. If the list of target parts is empty these processors will be applied to all message parts. Part indexes can be negative, and if so the part will be selected from the end counting backwards starting from -1. E.g. if part = -1 then the selected part will be the last part of the message, if part = -2 then the part before the last element will be selected, and so on. Some processors such as [`dedupe`](../dedupe/) act across an entire batch, when instead we might like to perform them on individual messages of a batch. In this case the [`for_each`](../for_each/) processor can be used. You can read more about batching [in this document](../../../configuration/batching/). --- # Page 208: archive **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/archive.md --- # archive --- title: archive latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/archive page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/archive.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/archive.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/archive/ "View the Cloud version of this component"), Self-Managed Archives all the messages of a batch into a single message according to the selected archive format. ```yml # Config fields, showing default values label: "" archive: format: "" # No default (required) path: "" ``` Some archive formats (such as tar, zip) treat each archive item (message part) as a file with a path. Since message parts only contain raw data a unique path must be generated for each part. This can be done by using function interpolations on the 'path' field as described in [Bloblang queries](../../../configuration/interpolation/#bloblang-queries). For types that aren’t file based (such as binary) the file field is ignored. The resulting archived message adopts the metadata of the _first_ message part of the batch. The functionality of this processor depends on being applied across messages that are batched. You can find out more about batching [in this doc](../../../configuration/batching/). ## [](#fields)Fields ### [](#format)`format` The archiving format to apply. **Type**: `string` | Option | Summary | | --- | --- | | binary | Archive messages to a binary blob format. | | concatenate | Join the raw contents of each message into a single binary message. | | json_array | Attempt to parse each message as a JSON document and append the result to an array, which becomes the contents of the resulting message. | | lines | Join the raw contents of each message and insert a line break between each one. | | tar | Archive messages to a unix standard tape archive. | | zip | Archive messages to a zip file. | ### [](#path)`path` The path to set for each message in the archive (when applicable). This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ```yaml # Examples: path: ${!count("files")}-${!timestamp_unix_nano()}.txt # --- path: ${!meta("kafka_key")}-${!json("id")}.json ``` ## [](#examples)Examples ### [](#tar-archive)Tar Archive If we had JSON messages in a batch each of the form: ```json {"doc":{"id":"foo","body":"hello world 1"}} ``` And we wished to tar archive them, setting their filenames to their respective unique IDs (with the extension `.json`), our config might look like this: ```yaml pipeline: processors: - archive: format: tar path: ${!json("doc.id")}.json ``` --- # Page 209: avro **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/avro.md --- # avro --- title: avro latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/avro page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/avro.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/avro.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Processor ▼ [Processor](/redpanda-connect/components/processors/avro/)[Scanner](/redpanda-connect/components/scanners/avro/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/avro/ "View the Cloud version of this component"), Self-Managed Performs Avro based operations on messages based on a schema. ```yml # Config fields, showing default values label: "" avro: operator: "" # No default (required) encoding: textual schema: "" schema_path: "" ``` > ⚠️ **WARNING** > > If you are consuming or generating messages using a schema registry service then it is likely this processor will fail as those services require messages to be prefixed with the identifier of the schema version being used. Instead, try the [`schema_registry_encode`](../schema_registry_encode/) and [`schema_registry_decode`](../schema_registry_decode/) processors. ## [](#operators)Operators ### [](#to_json)`to_json` Converts Avro documents into a JSON structure. This makes it easier to manipulate the contents of the document within Benthos. The encoding field specifies how the source documents are encoded. ### [](#from_json)`from_json` Attempts to convert JSON documents into Avro documents according to the specified encoding. ## [](#fields)Fields ### [](#encoding)`encoding` An Avro encoding format to use for conversions to and from a schema. **Type**: `string` **Default**: `textual` **Options**: `textual`, `binary`, `single` ### [](#operator)`operator` The [operator](#operators) to execute **Type**: `string` **Options**: `to_json`, `from_json` ### [](#schema)`schema` A full Avro schema to use. **Type**: `string` **Default**: `""` ### [](#schema_path)`schema_path` The path of a schema document to apply. Use either this or the `schema` field. URLs must begin with `file://` or `http://`. Note that `file://` URLs must use absolute paths (e.g. `[file:///absolute/path/to/spec.avsc](file:///absolute/path/to/spec.avsc)`); relative paths are not supported. **Type**: `string` **Default**: `""` ```yaml # Examples: schema_path: file:///path/to/spec.avsc # --- schema_path: http://localhost:8081/path/to/spec/versions/1 ``` --- # Page 210: awk **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/awk.md --- # awk --- title: awk latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/awk page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/awk.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/awk.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Available in:** Self-Managed Executes an AWK program on messages. This processor is very powerful as it offers a range of [custom functions](#awk-functions) for querying and mutating message contents and metadata. ```yml # Config fields, showing default values label: "" awk: codec: "" # No default (required) program: "" # No default (required) ``` Works by feeding message contents as the program input based on a chosen [codec](#codecs) and replaces the contents of each message with the result. If the result is empty (nothing is printed by the program) then the original message contents remain unchanged. Comes with a wide range of [custom functions](#awk-functions) for accessing message metadata, json fields, printing logs, etc. These functions can be overridden by functions within the program. Check out the [examples section](#examples) in order to see how this processor can be used. This processor uses [GoAWK](https://github.com/benhoyt/goawk), in order to understand the differences in how the program works you can read more about it in [goawk.differences](https://github.com/benhoyt/goawk#differences-from-awk). ## [](#fields)Fields ### [](#codec)`codec` A [codec](#codecs) defines how messages should be inserted into the AWK program as variables. The codec does not change which [custom Redpanda Connect functions](#awk-functions) are available. The `text` codec is the closest to a typical AWK use case. **Type**: `string` **Options**: `none`, `text`, `json` ### [](#program)`program` An AWK program to execute **Type**: `string` ## [](#examples)Examples ### [](#json-mapping-and-arithmetic)JSON Mapping and Arithmetic Because AWK is a full programming language it’s much easier to map documents and perform arithmetic with it than with other Redpanda Connect processors. For example, if we were expecting documents of the form: ```json {"doc":{"val1":5,"val2":10},"id":"1","type":"add"} {"doc":{"val1":5,"val2":10},"id":"2","type":"multiply"} ``` And we wished to perform the arithmetic specified in the `type` field, on the values `val1` and `val2` and, finally, map the result into the document, giving us the following resulting documents: ```json {"doc":{"result":15,"val1":5,"val2":10},"id":"1","type":"add"} {"doc":{"result":50,"val1":5,"val2":10},"id":"2","type":"multiply"} ``` We can do that with the following: ```yaml pipeline: processors: - awk: codec: none program: | function map_add_vals() { json_set_int("doc.result", json_get("doc.val1") + json_get("doc.val2")); } function map_multiply_vals() { json_set_int("doc.result", json_get("doc.val1") * json_get("doc.val2")); } function map_unknown(type) { json_set("error","unknown document type"); print_log("Document type not recognised: " type, "ERROR"); } { type = json_get("type"); if (type == "add") map_add_vals(); else if (type == "multiply") map_multiply_vals(); else map_unknown(type); } ``` ### [](#stuff-with-arrays)Stuff With Arrays It’s possible to iterate JSON arrays by appending an index value to the path, this can be used to do things like removing duplicates from arrays. For example, given the following input document: ```json {"path":{"to":{"foos":["one","two","three","two","four"]}}} ``` We could create a new array `foos_unique` from `foos` giving us the result: ```json {"path":{"to":{"foos":["one","two","three","two","four"],"foos_unique":["one","two","three","four"]}}} ``` With the following config: ```yaml pipeline: processors: - awk: codec: none program: | { array_path = "path.to.foos" array_len = json_length(array_path) for (i = 0; i < array_len; i++) { ele = json_get(array_path "." i) if ( ! ( ele in seen ) ) { json_append(array_path "_unique", ele) seen[ele] = 1 } } } ``` ## [](#codecs)Codecs The chosen codec determines how the contents of the message are fed into the program. Codecs only impact the input string and variables initialized for your program, they do not change the range of custom functions available. ### [](#none)`none` An empty string is fed into the program. Functions can still be used in order to extract and mutate metadata and message contents. This is useful for when your program only uses functions and doesn’t need the full text of the message to be parsed by the program, as it is significantly faster. ### [](#text)`text` The full contents of the message are fed into the program as a string, allowing you to reference tokenized segments of the message with variables ($0, $1, etc). Custom functions can still be used with this codec. This is the default codec as it behaves most similar to typical usage of the awk command line tool. ### [](#json)`json` An empty string is fed into the program, and variables are automatically initialized before execution of your program by walking the flattened JSON structure. Each value is converted into a variable by taking its full path, e.g. the object: ```json { "foo": { "bar": { "value": 10 }, "created_at": "2018-12-18T11:57:32" } } ``` Would result in the following variable declarations: ```none foo_bar_value = 10 foo_created_at = "2018-12-18T11:57:32" ``` Custom functions can also still be used with this codec. ## [](#awk-functions)AWK functions ### [](#json_get)`json_get` Signature: `json_get(path)` Attempts to find a JSON value in the input message payload by a [dot separated path](../../../configuration/field_paths/) and returns it as a string. ### [](#json_set)`json_set` Signature: `json_set(path, value)` Attempts to set a JSON value in the input message payload identified by a [dot separated path](../../../configuration/field_paths/), the value argument will be interpreted as a string. In order to set non-string values use one of the following typed varieties: - `json_set_int(path, value)` - `json_set_float(path, value)` - `json_set_bool(path, value)` ### [](#json_append)`json_append` Signature: `json_append(path, value)` Attempts to append a value to an array identified by a [dot separated path](../../../configuration/field_paths/). If the target does not exist it will be created. If the target exists but is not already an array then it will be converted into one, with its original contents set to the first element of the array. The value argument will be interpreted as a string. In order to append non-string values use one of the following typed varieties: - `json_append_int(path, value)` - `json_append_float(path, value)` - `json_append_bool(path, value)` ### [](#json_delete)`json_delete` Signature: `json_delete(path)` Attempts to delete a JSON field from the input message payload identified by a [dot separated path](../../../configuration/field_paths/). ### [](#json_length)`json_length` Signature: `json_length(path)` Returns the size of the string or array value of JSON field from the input message payload identified by a [dot separated path](../../../configuration/field_paths/). If the target field does not exist, or is not a string or array type, then zero is returned. In order to explicitly check the type of a field use `json_type`. ### [](#json_type)`json_type` Signature: `json_type(path)` Returns the type of a JSON field from the input message payload identified by a [dot separated path](../../../configuration/field_paths/). Possible values are: "string", "int", "float", "bool", "undefined", "null", "array", "object". ### [](#create_json_object)`create_json_object` Signature: `create_json_object(key1, val1, key2, val2, …​)` Generates a valid JSON object of key value pair arguments. The arguments are variadic, meaning any number of pairs can be listed. The value will always resolve to a string regardless of the value type. E.g. the following call: `create_json_object("a", "1", "b", 2, "c", "3")` Would result in this string: `\{"a":"1","b":"2","c":"3"}` ### [](#create_json_array)`create_json_array` Signature: `create_json_array(val1, val2, …​)` Generates a valid JSON array of value arguments. The arguments are variadic, meaning any number of values can be listed. The value will always resolve to a string regardless of the value type. E.g. the following call: `create_json_array("1", 2, "3")` Would result in this string: `["1","2","3"]` ### [](#metadata_set)`metadata_set` Signature: `metadata_set(key, value)` Set a metadata key for the message to a value. The value will always resolve to a string regardless of the value type. ### [](#metadata_get)`metadata_get` Signature: `metadata_get(key) string` Get the value of a metadata key from the message. ### [](#timestamp_unix)`timestamp_unix` Signature: `timestamp_unix() int` Returns the current unix timestamp (the number of seconds since 01-01-1970). ### [](#timestamp_unix-2)`timestamp_unix` Signature: `timestamp_unix(date) int` Attempts to parse a date string by detecting its format and returns the equivalent unix timestamp (the number of seconds since 01-01-1970). ### [](#timestamp_unix-3)`timestamp_unix` Signature: `timestamp_unix(date, format) int` Attempts to parse a date string according to a format and returns the equivalent unix timestamp (the number of seconds since 01-01-1970). The format is defined by showing how the reference time, defined to be `Mon Jan 2 15:04:05 -0700 MST 2006` would be displayed if it were the value. ### [](#timestamp_unix_nano)`timestamp_unix_nano` Signature: `timestamp_unix_nano() int` Returns the current unix timestamp in nanoseconds (the number of nanoseconds since 01-01-1970). ### [](#timestamp_unix_nano-2)`timestamp_unix_nano` Signature: `timestamp_unix_nano(date) int` Attempts to parse a date string by detecting its format and returns the equivalent unix timestamp in nanoseconds (the number of nanoseconds since 01-01-1970). ### [](#timestamp_unix_nano-3)`timestamp_unix_nano` Signature: `timestamp_unix_nano(date, format) int` Attempts to parse a date string according to a format and returns the equivalent unix timestamp in nanoseconds (the number of nanoseconds since 01-01-1970). The format is defined by showing how the reference time, defined to be `Mon Jan 2 15:04:05 -0700 MST 2006` would be displayed if it were the value. ### [](#timestamp_format)`timestamp_format` Signature: `timestamp_format(unix, format) string` Formats a unix timestamp. The format is defined by showing how the reference time, defined to be `Mon Jan 2 15:04:05 -0700 MST 2006` would be displayed if it were the value. The format is optional, and if omitted RFC3339 (`2006-01-02T15:04:05Z07:00`) will be used. ### [](#timestamp_format_nano)`timestamp_format_nano` Signature: `timestamp_format_nano(unixNano, format) string` Formats a unix timestamp in nanoseconds. The format is defined by showing how the reference time, defined to be `Mon Jan 2 15:04:05 -0700 MST 2006` would be displayed if it were the value. The format is optional, and if omitted RFC3339 (`2006-01-02T15:04:05Z07:00`) will be used. ### [](#print_log)`print_log` Signature: `print_log(message, level)` Prints a Redpanda Connect log message at a particular log level. The log level is optional, and if omitted the level `INFO` will be used. ### [](#base64_encode)`base64_encode` Signature: `base64_encode(data)` Encodes the input data to a base64 string. ### [](#base64_decode)`base64_decode` Signature: `base64_decode(data)` Attempts to base64-decode the input data and returns the decoded string if successful. It will emit an error otherwise. --- # Page 211: aws_bedrock_chat **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/aws_bedrock_chat.md --- # aws\_bedrock\_chat --- title: aws_bedrock_chat latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/aws_bedrock_chat page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/aws_bedrock_chat.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/aws_bedrock_chat.adoc page-git-created-date: "2024-09-04" page-git-modified-date: "2026-01-13" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/aws_bedrock_chat/ "View the Cloud version of this component"), Self-Managed Generates responses to messages in a chat conversation, using the [AWS Bedrock API](https://aws.amazon.com/bedrock/). Introduced in version 4.34.0. #### Common ```yml processors: label: "" aws_bedrock_chat: model: "" # No default (required) prompt: "" # No default (optional) system_prompt: "" # No default (optional) max_tokens: "" # No default (optional) temperature: "" # No default (optional) ``` #### Advanced ```yml processors: label: "" aws_bedrock_chat: region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) model: "" # No default (required) prompt: "" # No default (optional) system_prompt: "" # No default (optional) max_tokens: "" # No default (optional) temperature: "" # No default (optional) stop: [] # No default (optional) top_p: "" # No default (optional) ``` This processor sends prompts to your chosen large language model (LLM) and generates text from the responses, using the AWS Bedrock API. For more information, see the [AWS Bedrock documentation](https://docs.aws.amazon.com/bedrock/latest/userguide). ## [](#fields)Fields ### [](#credentials)`credentials` Configure which AWS credentials to use (optional). For more information, see [Amazon Web Services](../../../guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` The role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` The external ID to use when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials you want to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials you want to use. You must enter this value when using short-term credentials. **Type**: `string` ### [](#endpoint)`endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#max_tokens)`max_tokens` The maximum number of tokens to allow in the generated response. **Type**: `int` ### [](#model)`model` The model ID to use. For a full list, see the [AWS Bedrock documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html). **Type**: `string` ```yaml # Examples: model: amazon.titan-text-express-v1 # --- model: anthropic.claude-3-5-sonnet-20240620-v1:0 # --- model: cohere.command-text-v14 # --- model: meta.llama3-1-70b-instruct-v1:0 # --- model: mistral.mistral-large-2402-v1:0 ``` ### [](#prompt)`prompt` The prompt you want to generate a response for. By default, the processor submits the entire payload as a string. **Type**: `string` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#stop)`stop[]` A list of stop sequences. A stop sequence is a sequence of characters that causes the model to stop generating the response. **Type**: `array` ### [](#system_prompt)`system_prompt` The system prompt to submit to the AWS Bedrock LLM. **Type**: `string` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#temperature)`temperature` The likelihood of the model selecting higher-probability options while generating a response. A lower value makes the model more likely to choose higher-probability options. A higher value makes the model more likely to choose lower-probability options. **Type**: `float` ### [](#top_p)`top_p` The percentage of most-likely candidates that the model considers for the next token. For example, if you choose a value of `0.8`, the model selects from the top 80% of the probability distribution of tokens that could be next in the sequence. **Type**: `float` --- # Page 212: aws_bedrock_embeddings **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/aws_bedrock_embeddings.md --- # aws\_bedrock\_embeddings --- title: aws_bedrock_embeddings latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/aws_bedrock_embeddings page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/aws_bedrock_embeddings.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/aws_bedrock_embeddings.adoc page-git-created-date: "2024-10-16" page-git-modified-date: "2026-01-13" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/aws_bedrock_embeddings/ "View the Cloud version of this component"), Self-Managed Generates vector embeddings from text prompts, using the [AWS Bedrock API](https://aws.amazon.com/bedrock/). Introduced in version 4.37.0. #### Common ```yaml # Common config fields, showing default values label: "" aws_bedrock_embeddings: model: amazon.titan-embed-text-v1 # No default (required) text: "" # No default (optional) ``` #### Advanced ```yaml # All config fields, showing default values label: "" aws_bedrock_embeddings: region: "" endpoint: "" credentials: profile: "" id: "" secret: "" token: "" from_ec2_role: false role: "" role_external_id: "" model: amazon.titan-embed-text-v1 # No default (required) text: "" # No default (optional) ``` This processor sends text prompts to your chosen large language model (LLM), which generates vector embeddings for them using the AWS Bedrock API. For more information, see the [AWS Bedrock documentation](https://docs.aws.amazon.com/bedrock/latest/userguide). ## [](#fields)Fields ### [](#credentials)`credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](../../../guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of the AWS credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` The role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the AWS credentials in use. This is a required value for short-term credentials. **Type**: `string` ### [](#endpoint)`endpoint` A custom endpoint URL for AWS API requests. Use this to connect to AWS-compatible services or local testing environments instead of the standard AWS endpoints. **Type**: `string` ### [](#model)`model` The ID of the LLM that you want to use to generate vector embeddings. For a full list, see the [AWS Bedrock documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html). **Type**: `string` ```yaml # Examples: model: amazon.titan-embed-text-v1 # --- model: amazon.titan-embed-text-v2:0 # --- model: cohere.embed-english-v3 # --- model: cohere.embed-multilingual-v3 ``` ### [](#region)`region` The region in which your AWS resources are hosted. **Type**: `string` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#text)`text` The prompt you want to generate a vector embedding for. The processor submits the entire payload as a string. **Type**: `string` --- # Page 213: aws_dynamodb_partiql **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/aws_dynamodb_partiql.md --- # aws\_dynamodb\_partiql --- title: aws_dynamodb_partiql latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/aws_dynamodb_partiql page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/aws_dynamodb_partiql.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/aws_dynamodb_partiql.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/aws_dynamodb_partiql/ "View the Cloud version of this component"), Self-Managed Executes a PartiQL expression against a DynamoDB table for each message. Introduced in version 3.48.0. #### Common ```yml processors: label: "" aws_dynamodb_partiql: query: "" # No default (required) args_mapping: "" ``` #### Advanced ```yml processors: label: "" aws_dynamodb_partiql: query: "" # No default (required) unsafe_dynamic_query: false args_mapping: "" region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) ``` Both writes or reads are supported, when the query is a read the contents of the message will be replaced with the result. This processor is more efficient when messages are pre-batched as the whole batch will be executed in a single call. ## [](#examples)Examples ### [](#insert)Insert The following example inserts rows into the table footable with the columns foo, bar and baz populated with values extracted from messages: ```yaml pipeline: processors: - aws_dynamodb_partiql: query: "INSERT INTO footable VALUE {'foo':'?','bar':'?','baz':'?'}" args_mapping: | root = [ { "S": this.foo }, { "S": meta("kafka_topic") }, { "S": this.document.content }, ] ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` A [Bloblang mapping](../../../guides/bloblang/about/) that, for each message, creates a list of arguments to use with the query. **Type**: `string` **Default**: `""` ### [](#credentials)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](../../../guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#query)`query` A PartiQL query to execute for each message. **Type**: `string` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#tcp)`tcp` TCP socket configuration. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#unsafe_dynamic_query)`unsafe_dynamic_query` Whether to enable dynamic queries that support interpolation functions. **Type**: `bool` **Default**: `false` --- # Page 214: aws_lambda **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/aws_lambda.md --- # aws\_lambda --- title: aws_lambda latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/aws_lambda page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/aws_lambda.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/aws_lambda.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/aws_lambda/ "View the Cloud version of this component"), Self-Managed Invokes an AWS lambda for each message. The contents of the message is the payload of the request, and the result of the invocation will become the new contents of the message. Introduced in version 3.36.0. #### Common ```yml processors: label: "" aws_lambda: parallel: false function: "" # No default (required) ``` #### Advanced ```yml processors: label: "" aws_lambda: parallel: false function: "" # No default (required) rate_limit: "" region: "" # No default (optional) endpoint: "" # No default (optional) tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s credentials: profile: "" # No default (optional) id: "" # No default (optional) secret: "" # No default (optional) token: "" # No default (optional) from_ec2_role: "" # No default (optional) role: "" # No default (optional) role_external_id: "" # No default (optional) timeout: 5s retries: 3 ``` The `rate_limit` field can be used to specify a rate limit [resource](../../rate_limits/about/) to cap the rate of requests across parallel components service wide. In order to map or encode the payload to a specific request body, and map the response back into the original payload instead of replacing it entirely, you can use the [`branch` processor](../branch/). ## [](#error-handling)Error handling When Redpanda Connect is unable to connect to the AWS endpoint or is otherwise unable to invoke the target lambda function it will retry the request according to the configured number of retries. Once these attempts have been exhausted the failed message will continue through the pipeline with it’s contents unchanged, but flagged as having failed, allowing you to use [standard processor error handling patterns](../../../configuration/error_handling/). However, if the invocation of the function is successful but the function itself throws an error, then the message will have it’s contents updated with a JSON payload describing the reason for the failure, and a metadata field `lambda_function_error` will be added to the message allowing you to detect and handle function errors with a [`branch`](../branch/): ```yaml pipeline: processors: - branch: processors: - aws_lambda: function: foo result_map: | root = if meta().exists("lambda_function_error") { throw("Invocation failed due to %v: %v".format(this.errorType, this.errorMessage)) } else { this } output: switch: retry_until_success: false cases: - check: errored() output: reject: ${! error() } - output: resource: somewhere_else ``` ## [](#credentials)Credentials By default Redpanda Connect will use a shared credentials file when connecting to AWS services. It’s also possible to set them explicitly at the component level, allowing you to transfer data across accounts. You can find out more in [Amazon Web Services](../../../guides/cloud/aws/). ## [](#examples)Examples ### [](#branched-invoke)Branched Invoke This example uses a [`branch` processor](../branch/) to map a new payload for triggering a lambda function with an ID and username from the original message, and the result of the lambda is discarded, meaning the original message is unchanged. ```yaml pipeline: processors: - branch: request_map: '{"id":this.doc.id,"username":this.user.name}' processors: - aws_lambda: function: trigger_user_update ``` ## [](#fields)Fields ### [](#credentials-2)`credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](../../../guides/cloud/aws/). **Type**: `object` ### [](#credentials-from_ec2_role)`credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#credentials-id)`credentials.id` The ID of credentials to use. **Type**: `string` ### [](#credentials-profile)`credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#credentials-role)`credentials.role` A role ARN to assume. **Type**: `string` ### [](#credentials-role_external_id)`credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#credentials-secret)`credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#credentials-token)`credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#endpoint)`endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#function)`function` The function to invoke. **Type**: `string` ### [](#parallel)`parallel` Whether messages of a batch should be dispatched in parallel. **Type**: `bool` **Default**: `false` ### [](#rate_limit)`rate_limit` An optional [`rate_limit`](../../rate_limits/about/) to throttle invocations by. **Type**: `string` **Default**: `""` ### [](#region)`region` The AWS region to target. **Type**: `string` ### [](#retries)`retries` The maximum number of retry attempts for each message. **Type**: `int` **Default**: `3` ### [](#tcp)`tcp` TCP socket configuration. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` The maximum period of time to wait before abandoning an invocation. **Type**: `string` **Default**: `5s` --- # Page 215: azure_cosmosdb **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/azure_cosmosdb.md --- # azure\_cosmosdb --- title: azure_cosmosdb latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/azure_cosmosdb page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/azure_cosmosdb.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/azure_cosmosdb.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Processor ▼ [Processor](/redpanda-connect/components/processors/azure_cosmosdb/)[Input](/redpanda-connect/components/inputs/azure_cosmosdb/)[Output](/redpanda-connect/components/outputs/azure_cosmosdb/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/azure_cosmosdb/ "View the Cloud version of this component"), Self-Managed Creates or updates messages as JSON documents in [Azure CosmosDB](https://learn.microsoft.com/en-us/azure/cosmos-db/introduction). Introduced in version v4.25.0. #### Common ```yml processors: label: "" azure_cosmosdb: endpoint: "" # No default (optional) account_key: "" # No default (optional) connection_string: "" # No default (optional) database: "" # No default (required) container: "" # No default (required) partition_keys_map: "" # No default (required) operation: Create item_id: "" # No default (optional) ``` #### Advanced ```yml processors: label: "" azure_cosmosdb: endpoint: "" # No default (optional) account_key: "" # No default (optional) connection_string: "" # No default (optional) database: "" # No default (required) container: "" # No default (required) partition_keys_map: "" # No default (required) operation: Create patch_operations: [] # No default (optional) patch_condition: "" # No default (optional) auto_id: true item_id: "" # No default (optional) enable_content_response_on_write: true ``` When creating documents, each message must have the `id` property (case-sensitive) set (or use `auto_id: true`). It is the unique name that identifies the document, that is, no two documents share the same `id` within a logical partition. The `id` field must not exceed 255 characters. [See details](https://learn.microsoft.com/en-us/rest/api/cosmos-db/documents). The `partition_keys` field must resolve to the same value(s) across the entire message batch. ## [](#credentials)Credentials You can use one of the following authentication mechanisms: - Set the `endpoint` field and the `account_key` field - Set only the `endpoint` field to use [DefaultAzureCredential](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/azidentity#DefaultAzureCredential) - Set the `connection_string` field ## [](#metadata)Metadata This component adds the following metadata fields to each message: ```none - activity_id - request_charge ``` You can access these metadata fields using [function interpolation](../../../configuration/interpolation/#bloblang-queries). ## [](#batching)Batching CosmosDB limits the maximum batch size to 100 messages and the payload must not exceed 2MB ([details here](https://learn.microsoft.com/en-us/azure/cosmos-db/concepts-limits#per-request-limits)). ## [](#examples)Examples ### [](#patch-documents)Patch documents Query documents from a container and patch them. ```yaml input: azure_cosmosdb: endpoint: http://localhost:8080 account_key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw== database: blobbase container: blobfish partition_keys_map: root = "AbyssalPlain" query: SELECT * FROM blobfish processors: - mapping: | root = "" meta habitat = json("habitat") meta id = this.id - azure_cosmosdb: endpoint: http://localhost:8080 account_key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw== database: testdb container: blobfish partition_keys_map: root = json("habitat") item_id: ${! meta("id") } operation: Patch patch_operations: # Add a new /diet field - operation: Add path: /diet value_map: root = json("diet") # Remove the first location from the /locations array field - operation: Remove path: /locations/0 # Add new location at the end of the /locations array field - operation: Add path: /locations/- value_map: root = "Challenger Deep" # Return the updated document enable_content_response_on_write: true ``` ## [](#fields)Fields ### [](#account_key)`account_key` Account key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ```yaml # Examples: account_key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw== ``` ### [](#auto_id)`auto_id` Automatically set the item `id` field to a random UUID v4. If the `id` field is already set, then it will not be overwritten. Setting this to `false` can improve performance, since the messages will not have to be parsed. **Type**: `bool` **Default**: `true` ### [](#connection_string)`connection_string` Connection string. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ```yaml # Examples: connection_string: AccountEndpoint=https://localhost:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==; ``` ### [](#container)`container` Container. **Type**: `string` ```yaml # Examples: container: testcontainer ``` ### [](#database)`database` Database. **Type**: `string` ```yaml # Examples: database: testdb ``` ### [](#enable_content_response_on_write)`enable_content_response_on_write` Enable content response on write operations. To save some bandwidth, set this to false if you don’t need to receive the updated message(s) from the server, in which case the processor will not modify the content of the messages which are fed into it. Applies to every operation except Read. **Type**: `bool` **Default**: `true` ### [](#endpoint)`endpoint` CosmosDB endpoint. **Type**: `string` ```yaml # Examples: endpoint: https://localhost:8081 ``` ### [](#item_id)`item_id` ID of item to replace or delete. Only used by the Replace and Delete operations This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: item_id: ${! json("id") } ``` ### [](#operation)`operation` Operation. **Type**: `string` **Default**: `Create` | Option | Summary | | --- | --- | | Create | Create operation. | | Delete | Delete operation. | | Patch | Patch operation. | | Read | Read operation. | | Replace | Replace operation. | | Upsert | Upsert operation. | ### [](#partition_keys_map)`partition_keys_map` A [Bloblang mapping](../../../guides/bloblang/about/) which should evaluate to a single partition key value or an array of partition key values of type string, integer or boolean. Currently, hierarchical partition keys are not supported so only one value may be provided. **Type**: `string` ```yaml # Examples: partition_keys_map: root = "blobfish" # --- partition_keys_map: root = 41 # --- partition_keys_map: root = true # --- partition_keys_map: root = null # --- partition_keys_map: root = json("blobfish").depth ``` ### [](#patch_condition)`patch_condition` Patch operation condition. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: patch_condition: from c where not is_defined(c.blobfish) ``` ### [](#patch_operations)`patch_operations[]` Patch operations to be performed when `operation: Patch` . **Type**: `object` ### [](#patch_operations-operation)`patch_operations[].operation` Operation. **Type**: `string` **Default**: `Add` | Option | Summary | | --- | --- | | Add | Add patch operation. | | Increment | Increment patch operation. | | Remove | Remove patch operation. | | Replace | Replace patch operation. | | Set | Set patch operation. | ### [](#patch_operations-path)`patch_operations[].path` Path. **Type**: `string` ```yaml # Examples: path: /foo/bar/baz ``` ### [](#patch_operations-value_map)`patch_operations[].value_map` A [Bloblang mapping](../../../guides/bloblang/about/) which should evaluate to a value of any type that is supported by CosmosDB. **Type**: `string` ```yaml # Examples: value_map: root = "blobfish" # --- value_map: root = 41 # --- value_map: root = true # --- value_map: root = json("blobfish").depth # --- value_map: root = [1, 2, 3] ``` ## [](#cosmosdb-emulator)CosmosDB emulator If you wish to run the CosmosDB emulator that is referenced in the documentation [here](https://learn.microsoft.com/en-us/azure/cosmos-db/linux-emulator), the following Docker command should do the trick: ```bash > docker run --rm -it -p 8081:8081 --name=cosmosdb -e AZURE_COSMOS_EMULATOR_PARTITION_COUNT=10 -e AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE=false mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator ``` Note: `AZURE_COSMOS_EMULATOR_PARTITION_COUNT` controls the number of partitions that will be supported by the emulator. The bigger the value, the longer it takes for the container to start up. Additionally, instead of installing the container self-signed certificate which is exposed via `[https://localhost:8081/_explorer/emulator.pem](https://localhost:8081/_explorer/emulator.pem)`, you can run [mitmproxy](https://mitmproxy.org/) like so: ```bash > mitmproxy -k --mode "reverse:https://localhost:8081" ``` Then you can access the CosmosDB UI via `[http://localhost:8080/_explorer/index.html](http://localhost:8080/_explorer/index.html)` and use `[http://localhost:8080](http://localhost:8080)` as the CosmosDB endpoint. --- # Page 216: benchmark **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/benchmark.md --- # benchmark --- title: benchmark latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/benchmark page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/benchmark.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/benchmark.adoc page-git-created-date: "2024-12-16" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/benchmark/ "View the Cloud version of this component"), Self-Managed Logs throughput statistics for processed messages, and provides a summary of those statistics over the lifetime of the processor. Introduced in version 4.40.0. ```yml # Configuration fields, showing default values label: "" benchmark: interval: 5s count_bytes: true ``` ## [](#throughput-statistics)Throughput statistics This processor logs the following rolling statistics at a [configurable interval](#interval) to help you to understand the current performance of your pipeline: - The number of messages processed per second. - The number of bytes processed per second (optional). For example: ```bash INFO rolling stats: 1 msg/sec, 407 B/sec ``` When the processor shuts down, it also logs a summary of the number and size of messages processed during its lifetime. For example: ```bash INFO total stats: 1.00186 msg/sec, 425 B/sec ``` ## [](#fields)Fields ### [](#count_bytes)`count_bytes` Whether to measure the number of bytes per second of throughput. If set to `true`, Redpanda Connect must serialize structured data to count the number of bytes processed, which can unnecessarily degrade performance if serialization is not required elsewhere in your pipeline. **Type**: `bool` **Default**: `true` ### [](#interval)`interval` How often to emit rolling statistics. Set to `0`, if you only want to log summary statistics when the processor shuts down. **Type**: `string` **Default**: `5s` --- # Page 217: bloblang **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/bloblang.md --- # bloblang --- title: bloblang latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/bloblang page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/bloblang.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/bloblang.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/bloblang/ "View the Cloud version of this component"), Self-Managed Executes a [Bloblang](../../../guides/bloblang/about/) mapping on messages. ```yml # Config fields, showing default values label: "" bloblang: "" ``` Bloblang is a powerful language that enables a wide range of mapping, transformation and filtering tasks. For more information see [Bloblang](../../../guides/bloblang/about/). If your mapping is large and you’d prefer for it to live in a separate file then you can execute a mapping directly from a file with the expression `from ""`, where the path must be absolute, or relative from the location that Redpanda Connect is executed from. ## [](#component-rename)Component rename This processor was recently renamed to the [`mapping` processor](../mapping/) in order to make the purpose of the processor more prominent. It is still valid to use the existing `bloblang` name but eventually it will be deprecated and replaced by the new name in example configs. ## [](#examples)Examples ### [](#mapping)Mapping Given JSON documents containing an array of fans: ```json { "id":"foo", "description":"a show about foo", "fans":[ {"name":"bev","obsession":0.57}, {"name":"grace","obsession":0.21}, {"name":"ali","obsession":0.89}, {"name":"vic","obsession":0.43} ] } ``` We can reduce the fans to only those with an obsession score above 0.5, giving us: ```json { "id":"foo", "description":"a show about foo", "fans":[ {"name":"bev","obsession":0.57}, {"name":"ali","obsession":0.89} ] } ``` With the following config: ```yaml pipeline: processors: - bloblang: | root = this root.fans = this.fans.filter(fan -> fan.obsession > 0.5) ``` ### [](#more-mapping)More Mapping When receiving JSON documents of the form: ```json { "locations": [ {"name": "Seattle", "state": "WA"}, {"name": "New York", "state": "NY"}, {"name": "Bellevue", "state": "WA"}, {"name": "Olympia", "state": "WA"} ] } ``` We could collapse the location names from the state of Washington into a field `Cities`: ```json {"Cities": "Bellevue, Olympia, Seattle"} ``` With the following config: ```yaml pipeline: processors: - bloblang: | root.Cities = this.locations. filter(loc -> loc.state == "WA"). map_each(loc -> loc.name). sort().join(", ") ``` ## [](#error-handling)Error handling Bloblang mappings can fail, in which case the message remains unchanged, errors are logged, and the message is flagged as having failed, allowing you to use [standard processor error handling patterns](../../../configuration/error_handling/). However, Bloblang itself also provides powerful ways of ensuring your mappings do not fail by specifying desired fallback behavior, which you can read about in [Error handling](../../../guides/bloblang/about/#error-handling.adoc). --- # Page 218: bounds_check **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/bounds_check.md --- # bounds\_check --- title: bounds_check latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/bounds_check page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/bounds_check.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/bounds_check.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/bounds_check/ "View the Cloud version of this component"), Self-Managed Removes messages (and batches) that do not fit within certain size boundaries. #### Common ```yml processors: label: "" bounds_check: max_part_size: 1073741824 min_part_size: 1 ``` #### Advanced ```yml processors: label: "" bounds_check: max_part_size: 1073741824 min_part_size: 1 max_parts: 100 min_parts: 1 ``` ## [](#fields)Fields ### [](#max_part_size)`max_part_size` The maximum size of a message to allow (in bytes) **Type**: `int` **Default**: `1073741824` ### [](#max_parts)`max_parts` The maximum size of message batches to allow (in message count) **Type**: `int` **Default**: `100` ### [](#min_part_size)`min_part_size` The minimum size of a message to allow (in bytes) **Type**: `int` **Default**: `1` ### [](#min_parts)`min_parts` The minimum size of message batches to allow (in message count) **Type**: `int` **Default**: `1` --- # Page 219: branch **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/branch.md --- # branch --- title: branch latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/branch page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/branch.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/branch.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/branch/ "View the Cloud version of this component"), Self-Managed The `branch` processor allows you to create a new request message via a [Bloblang mapping](../../../guides/bloblang/about/), execute a list of processors on the request messages, and, finally, map the result back into the source message using another mapping. ```yml # Config fields, showing default values label: "" branch: request_map: "" processors: [] # No default (required) result_map: "" ``` This is useful for preserving the original message contents when using processors that would otherwise replace the entire contents. ## [](#metadata)Metadata Metadata fields that are added to messages during branch processing will not be automatically copied into the resulting message. In order to do this you should explicitly declare in your `result_map` either a wholesale copy with `meta = metadata()`, or selective copies with `meta foo = metadata("bar")` and so on. It is also possible to reference the metadata of the origin message in the `result_map` using the [`@` operator](../../../guides/bloblang/about/#metadata). ## [](#error-handling)Error handling If the `request_map` fails the child processors will not be executed. If the child processors themselves result in an (uncaught) error then the `result_map` will not be executed. If the `result_map` fails the message will remain unchanged. Under any of these conditions standard [error handling methods](../../../configuration/error_handling/) can be used in order to filter, DLQ or recover the failed messages. ## [](#conditional-branching)Conditional branching If the root of your request map is set to `deleted()` then the branch processors are skipped for the given message, this allows you to conditionally branch messages. ## [](#fields)Fields ### [](#processors)`processors[]` A list of processors to apply to mapped requests. When processing message batches the resulting batch must match the size and ordering of the input batch, therefore filtering, grouping should not be performed within these processors. **Type**: `processor` ### [](#request_map)`request_map` A [Bloblang mapping](../../../guides/bloblang/about/) that describes how to create a request payload suitable for the child processors of this branch. If left empty then the branch will begin with an exact copy of the origin message (including metadata). **Type**: `string` **Default**: `""` ```yaml # Examples: request_map: |- root = { "id": this.doc.id, "content": this.doc.body.text } # --- request_map: |- root = if this.type == "foo" { this.foo.request } else { deleted() } ``` ### [](#result_map)`result_map` A [Bloblang mapping](../../../guides/bloblang/about/) that describes how the resulting messages from branched processing should be mapped back into the original payload. If left empty the origin message will remain unchanged (including metadata). **Type**: `string` **Default**: `""` ```yaml # Examples: result_map: |- meta foo_code = metadata("code") root.foo_result = this # --- result_map: |- meta = metadata() root.bar.body = this.body root.bar.id = this.user.id # --- result_map: root.raw_result = content().string() # --- result_map: |- root.enrichments.foo = if metadata("request_failed") != null { throw(metadata("request_failed")) } else { this } # --- result_map: |- # Retain only the updated metadata fields which were present in the origin message meta = metadata().filter(v -> @.get(v.key) != null) ``` ## [](#examples)Examples ### [](#http-request)HTTP Request This example strips the request message into an empty body, grabs an HTTP payload, and places the result back into the original message at the path `image.pull_count`: ```yaml pipeline: processors: - branch: request_map: 'root = ""' processors: - http: url: https://hub.docker.com/v2/repositories/jeffail/benthos verb: GET headers: Content-Type: application/json result_map: root.image.pull_count = this.pull_count # Example input: {"id":"foo","some":"pre-existing data"} # Example output: {"id":"foo","some":"pre-existing data","image":{"pull_count":1234}} ``` ### [](#non-structured-results)Non Structured Results When the result of your branch processors is unstructured and you wish to simply set a resulting field to the raw output use the content function to obtain the raw bytes of the resulting message and then coerce it into your value type of choice: ```yaml pipeline: processors: - branch: request_map: 'root = this.document.id' processors: - cache: resource: descriptions_cache key: ${! content() } operator: get result_map: root.document.description = content().string() # Example input: {"document":{"id":"foo","content":"hello world"}} # Example output: {"document":{"id":"foo","content":"hello world","description":"this is a cool doc"}} ``` ### [](#lambda-function)Lambda Function This example maps a new payload for triggering a lambda function with an ID and username from the original message, and the result of the lambda is discarded, meaning the original message is unchanged. ```yaml pipeline: processors: - branch: request_map: '{"id":this.doc.id,"username":this.user.name}' processors: - aws_lambda: function: trigger_user_update # Example input: {"doc":{"id":"foo","body":"hello world"},"user":{"name":"fooey"}} # Output matches the input, which is unchanged ``` ### [](#conditional-caching)Conditional Caching This example caches a document by a message ID only when the type of the document is a foo: ```yaml pipeline: processors: - branch: request_map: | meta id = this.id root = if this.type == "foo" { this.document } else { deleted() } processors: - cache: resource: TODO operator: set key: ${! @id } value: ${! content() } ``` --- # Page 220: cache **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/cache.md --- # cache --- title: cache latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/cache page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/cache.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/cache.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Processor ▼ [Processor](/redpanda-connect/components/processors/cache/)[Output](/redpanda-connect/components/outputs/cache/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/cache/ "View the Cloud version of this component"), Self-Managed Performs operations against a [cache resource](../../caches/about/) for each message, allowing you to store or retrieve data within message payloads. #### Common ```yml processors: label: "" cache: resource: "" # No default (required) operator: "" # No default (required) key: "" # No default (required) value: "" # No default (optional) ``` #### Advanced ```yml processors: label: "" cache: resource: "" # No default (required) operator: "" # No default (required) key: "" # No default (required) value: "" # No default (optional) ttl: "" # No default (optional) ``` For use cases where you wish to cache the result of processors, consider using the [`cached` processor](../cached/) instead. This processor will interpolate functions within the `key` and `value` fields individually for each message. This allows you to specify dynamic keys and values based on the contents of the message payloads and metadata. You can find a list of functions in [Bloblang queries](../../../configuration/interpolation/#bloblang-queries). ## [](#examples)Examples ### [](#deduplication)Deduplication Deduplication can be done using the add operator with a key extracted from the message payload, since it fails when a key already exists we can remove the duplicates using a [`mapping` processor](../mapping/): ```yaml pipeline: processors: - cache: resource: foocache operator: add key: '${! json("message.id") }' value: "storeme" - mapping: root = if errored() { deleted() } cache_resources: - label: foocache redis: url: tcp://TODO:6379 ``` ### [](#deduplication-batch-wide)Deduplication Batch-Wide Sometimes it’s necessary to deduplicate a batch of messages (also known as a window) by a single identifying value. This can be done by introducing a [`branch` processor](../branch/), which executes the cache only once on behalf of the batch, in this case with a value make from a field extracted from the first and last messages of the batch: ```yaml pipeline: processors: # Try and add one message to a cache that identifies the whole batch - branch: request_map: | root = if batch_index() == 0 { json("id").from(0) + json("meta.tail_id").from(-1) } else { deleted() } processors: - cache: resource: foocache operator: add key: ${! content() } value: t # Delete all messages if we failed - mapping: | root = if errored().from(0) { deleted() } ``` ### [](#hydration)Hydration It’s possible to enrich payloads with content previously stored in a cache by using the [`branch`](../branch/) processor: ```yaml pipeline: processors: - branch: processors: - cache: resource: foocache operator: get key: '${! json("message.document_id") }' result_map: 'root.message.document = this' # NOTE: If the data stored in the cache is not valid JSON then use # something like this instead: # result_map: 'root.message.document = content().string()' cache_resources: - label: foocache memcached: addresses: [ "TODO:11211" ] ``` ## [](#fields)Fields ### [](#key)`key` A key to use with the cache. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#operator)`operator` The [operation](#operators) to perform with the cache. **Type**: `string` **Options**: `set`, `add`, `get`, `delete`, `exists` ### [](#resource)`resource` The [`cache` resource](../../caches/about/) to target with this processor. **Type**: `string` ### [](#ttl)`ttl` The time to live (TTL) of each individual item as a duration string. After this period an item will be eligible for removal during the next compaction. Not all caches support per-key TTLs, those that do will have a configuration field `default_ttl`, and those that do not will fall back to their generally configured TTL setting. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). Requires version 3.33.0 or later. **Type**: `string` ```yaml # Examples: ttl: 60s # --- ttl: 5m # --- ttl: 36h ``` ### [](#value)`value` A value to use with the cache (when applicable). This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ## [](#operators)Operators ### [](#set)`set` Set a key in the cache to a value. If the key already exists the contents are overridden. ### [](#add)`add` Set a key in the cache to a value. If the key already exists the action fails with a 'key already exists' error, which can be detected with [processor error handling](../../../configuration/error_handling/). ### [](#get)`get` Retrieve the contents of a cached key and replace the original message payload with the result. If the key does not exist the action fails with an error, which can be detected with [processor error handling](../../../configuration/error_handling/). ### [](#exists)`exists` Check whether a specific key is in the cache and replace the original message payload with `true` if the key exists, or `false` if it doesn’t. ### [](#delete)`delete` Delete a key and its contents from the cache. If the key does not exist the action is a no-op and will not fail with an error. --- # Page 221: cached **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/cached.md --- # cached --- title: cached latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/cached page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/cached.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/cached.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/cached/ "View the Cloud version of this component"), Self-Managed Cache the result of applying one or more processors to messages identified by a key. If the key already exists within the cache the contents of the message will be replaced with the cached result instead of applying the processors. This component is therefore useful in situations where an expensive set of processors need only be executed periodically. Introduced in version 4.3.0. ```yml # Config fields, showing default values label: "" cached: cache: "" # No default (required) skip_on: errored() # No default (optional) key: my_foo_result # No default (required) ttl: "" # No default (optional) processors: [] # No default (required) ``` The format of the data when stored within the cache is a custom and versioned schema chosen to balance performance and storage space. It is therefore not possible to point this processor to a cache that is pre-populated with data that this processor has not created itself. ## [](#examples)Examples ### [](#cached-enrichment)Cached Enrichment In the following example we want to we enrich messages consumed from Kafka with data specific to the origin topic partition, we do this by placing an `http` processor within a `branch`, where the HTTP URL contains interpolation functions with the topic and partition in the path. However, it would be inefficient to make this HTTP request for every single message as the result is consistent for all data of a given topic partition. We can solve this by placing our enrichment call within a `cached` processor where the key contains the topic and partition, resulting in messages that originate from the same topic/partition combination using the cached result of the prior. ```yaml pipeline: processors: - branch: processors: - cached: key: '${! meta("kafka_topic") }-${! meta("kafka_partition") }' cache: foo_cache processors: - mapping: 'root = ""' - http: url: http://example.com/enrichment/${! meta("kafka_topic") }/${! meta("kafka_partition") } verb: GET result_map: 'root.enrichment = this' cache_resources: - label: foo_cache memory: # Disable compaction so that cached items never expire compaction_interval: "" ``` ### [](#periodic-global-enrichment)Periodic Global Enrichment In the following example we enrich all messages with the same data obtained from a static URL with an `http` processor within a `branch`. However, we expect the data from this URL to change roughly every 10 minutes, so we configure a `cached` processor with a static key (since this request is consistent for all messages) and a TTL of `10m`. ```yaml pipeline: processors: - branch: request_map: 'root = ""' processors: - cached: key: static_foo cache: foo_cache ttl: 10m processors: - http: url: http://example.com/get/foo.json verb: GET result_map: 'root.foo = this' cache_resources: - label: foo_cache memory: {} ``` ## [](#fields)Fields ### [](#cache)`cache` The cache resource to read and write processor results from. **Type**: `string` ### [](#key)`key` A key to be resolved for each message, if the key already exists in the cache then the cached result is used, otherwise the processors are applied and the result is cached under this key. The key could be static and therefore apply generally to all messages or it could be an interpolated expression that is potentially unique for each message. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: key: my_foo_result # --- key: ${! this.document.id } # --- key: ${! meta("kafka_key") } # --- key: ${! meta("kafka_topic") } ``` ### [](#processors)`processors[]` The list of processors whose result will be cached. **Type**: `processor` ### [](#skip_on)`skip_on` A condition that can be used to skip caching the results from the processors. **Type**: `string` ```yaml # Examples: skip_on: errored() ``` ### [](#ttl)`ttl` An optional expiry period to set for each cache entry. Some caches only have a general TTL and will therefore ignore this setting. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 222: catch **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/catch.md --- # catch --- title: catch latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/catch page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/catch.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/catch.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/catch/ "View the Cloud version of this component"), Self-Managed Applies a list of child processors _only_ when a previous processing step has failed. ```yml # Config fields, showing default values label: "" catch: [] ``` Behaves similarly to the [`for_each`](../for_each/) processor, where a list of child processors are applied to individual messages of a batch. However, processors are only applied to messages that failed a processing step prior to the catch. For example, with the following config: ```yaml pipeline: processors: - resource: foo - catch: - resource: bar - resource: baz ``` If the processor `foo` fails for a particular message, that message will be fed into the processors `bar` and `baz`. Messages that do not fail for the processor `foo` will skip these processors. When messages leave the catch block their fail flags are cleared. This processor is useful for when it’s possible to recover failed messages, or when special actions (such as logging/metrics) are required before dropping them. More information about error handling can be found in [Error Handling](../../../configuration/error_handling/). --- # Page 223: cohere_chat **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/cohere_chat.md --- # cohere\_chat --- title: cohere_chat latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/cohere_chat page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/cohere_chat.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/cohere_chat.adoc page-git-created-date: "2024-10-16" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/cohere_chat/ "View the Cloud version of this component"), Self-Managed Generates responses to messages in a chat conversation, using the [Cohere API](https://docs.cohere.com/docs/chat-api) and external tools. Introduced in version 4.37.0. #### Common ```yml processors: label: "" cohere_chat: base_url: https://api.cohere.com api_key: "" # No default (required) model: "" # No default (required) prompt: "" # No default (optional) system_prompt: "" # No default (optional) max_tokens: "" # No default (optional) temperature: "" # No default (optional) response_format: text json_schema: "" # No default (optional) max_tool_calls: 10 tools: [] ``` #### Advanced ```yml processors: label: "" cohere_chat: base_url: https://api.cohere.com api_key: "" # No default (required) model: "" # No default (required) prompt: "" # No default (optional) system_prompt: "" # No default (optional) max_tokens: "" # No default (optional) temperature: "" # No default (optional) response_format: text json_schema: "" # No default (optional) schema_registry: url: "" # No default (required) subject: "" # No default (required) refresh_interval: "" # No default (optional) tls: skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} top_p: "" # No default (optional) frequency_penalty: "" # No default (optional) presence_penalty: "" # No default (optional) seed: "" # No default (optional) stop: [] # No default (optional) max_tool_calls: 10 tools: [] ``` This processor sends the contents of user prompts to the Cohere API, which generates responses using all available context, including supplementary data provided by external tools. By default, the processor submits the entire payload of each message as a string, unless you use the `prompt` field to customize it. To learn more about chat completion, see the [Cohere API documentation](https://docs.cohere.com/docs/chat-api). ## [](#fields)Fields ### [](#api_key)`api_key` The API key for the Cohere API. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#base_url)`base_url` The base URL to use for API requests. **Type**: `string` **Default**: `[https://api.cohere.com](https://api.cohere.com)` ### [](#frequency_penalty)`frequency_penalty` A number between `-2.0` and `2.0`. Positive values penalize new tokens based on the frequency of their appearance in the text so far. This decreases the model’s likelihood to repeat the same line verbatim. **Type**: `float` ### [](#json_schema)`json_schema` The JSON schema to use when responding in `json_schema` format. To learn more about the JSON schema features supported, see the [Cohere documentation](https://docs.cohere.com/docs/structured-outputs-json). **Type**: `string` ### [](#max_tokens)`max_tokens` The maximum number of tokens to allow in the chat completion. **Type**: `int` ### [](#max_tool_calls)`max_tool_calls` The maximum number of tool calls the model can perform. **Type**: `int` **Default**: `10` ### [](#model)`model` The name of the Cohere large language model (LLM) you want to use. **Type**: `string` ```yaml # Examples: model: command-r-plus # --- model: command-r # --- model: command # --- model: command-light ``` ### [](#presence_penalty)`presence_penalty` A number between `-2.0` and `2.0`. Positive values penalize new tokens based on the frequency of their appearance in the text so far. This increases the model’s likelihood to talk about new topics. **Type**: `float` ### [](#prompt)`prompt` The user prompt you want to generate a response for. By default, the processor submits the entire payload as a string. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#response_format)`response_format` Choose the model’s output format. If `json_schema` is specified, then you must also configure a `json_schema` or `schema_registry`. **Type**: `string` **Default**: `text` **Options**: `text`, `json`, `json_schema` ### [](#schema_registry)`schema_registry` The schema registry to dynamically load schemas from when responding in `json_schema` format. Schemas themselves must be in JSON format. To learn more about the JSON schema features supported, see the [Cohere documentation](https://docs.cohere.com/docs/structured-outputs-json). **Type**: `object` ### [](#schema_registry-basic_auth)`schema_registry.basic_auth` Configure basic authentication for requests from this component to your schema registry. **Type**: `object` ### [](#schema_registry-basic_auth-enabled)`schema_registry.basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-basic_auth-password)`schema_registry.basic_auth.password` The password to use for authentication. Used together with `username` for basic authentication or with encrypted private keys for secure access. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-basic_auth-username)`schema_registry.basic_auth.username` The username of the account credentials to authenticate as. Used together with `password` for basic authentication. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt)`schema_registry.jwt` Beta Configure JSON Web Token (JWT) authentication for secure data transmission from your schema registry to this component. This feature is in beta and may change in future releases. **Type**: `object` ### [](#schema_registry-jwt-claims)`schema_registry.jwt.claims` Values used to pass the identity of the authenticated entity to the service provider. In this case, between this component and the schema registry. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-enabled)`schema_registry.jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-jwt-headers)`schema_registry.jwt.headers` The key/value pairs that identify the type of token and signing algorithm. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-private_key_file)`schema_registry.jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt-signing_method)`schema_registry.jwt.signing_method` The cryptographic algorithm used to sign the JWT token. Supported algorithms include RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth)`schema_registry.oauth` Configure OAuth version 1.0 to give this component authorized access to your schema registry. **Type**: `object` ### [](#schema_registry-oauth-access_token)`schema_registry.oauth.access_token` The value this component can use to gain access to the data in the schema registry. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-access_token_secret)`schema_registry.oauth.access_token_secret` The secret that establishes ownership of the `oauth.access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_key)`schema_registry.oauth.consumer_key` The value used to identify this component or client to your schema registry. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_secret)`schema_registry.oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-enabled)`schema_registry.oauth.enabled` Whether to enable OAuth version 1.0 authentication for requests to the schema registry. **Type**: `bool` **Default**: `false` ### [](#schema_registry-refresh_interval)`schema_registry.refresh_interval` The refresh rate for fetching the latest schema. If not specified the schema does not refresh. **Type**: `string` ### [](#schema_registry-subject)`schema_registry.subject` The subject name to fetch the schema for. **Type**: `string` ### [](#schema_registry-tls)`schema_registry.tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#schema_registry-tls-client_certs)`schema_registry.tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#schema_registry-tls-client_certs-cert)`schema_registry.tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-cert_file)`schema_registry.tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key)`schema_registry.tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key_file)`schema_registry.tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-password)`schema_registry.tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#schema_registry-tls-enable_renegotiation)`schema_registry.tls.enable_renegotiation` Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-root_cas)`schema_registry.tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#schema_registry-tls-root_cas_file)`schema_registry.tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#schema_registry-tls-skip_cert_verify)`schema_registry.tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#schema_registry-url)`schema_registry.url` The base URL of the schema registry service. **Type**: `string` ### [](#seed)`seed` If specified, Redpanda Connect makes a best effort to sample deterministically. Repeated requests with the same seed and parameters should return the same result. Determinism is not guaranteed. **Type**: `int` ### [](#stop)`stop[]` Specify up to four sequences to stop the API from generating further tokens. **Type**: `array` ### [](#system_prompt)`system_prompt` The system prompt to submit along with the user prompt. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#temperature)`temperature` Choose a sampling temperature between `0` and `2`: - Higher values, such as `0.8` make the output more random. - Lower values, such as `0.2` make the output more focused and deterministic. Redpanda recommends adding a value for this field or `top_p`, but not both. **Type**: `float` ### [](#tools)`tools[]` External tools that the model can invoke, such as functions, APIs, or web browsing. You can define a series of processors that describe these tools, enabling the model to use agent-like behavior to decide when and how to invoke them to enhance response generation. **Type**: `object` **Default**: `[]` ### [](#tools-description)`tools[].description` A description of this tool, the LLM uses this to decide if the tool should be used. **Type**: `string` ### [](#tools-name)`tools[].name` The name of this tool. **Type**: `string` ### [](#tools-parameters)`tools[].parameters` The parameters the LLM needs to provide to invoke this tool. **Type**: `object` ### [](#tools-parameters-properties)`tools[].parameters.properties` The properties for the processor’s input data **Type**: `object` ### [](#tools-parameters-properties-description)`tools[].parameters.properties.description` A description of this parameter. **Type**: `string` ### [](#tools-parameters-properties-enum)`tools[].parameters.properties.enum[]` Specifies that this parameter is an enum and only these specific values should be used. **Type**: `array` **Default**: `[]` ### [](#tools-parameters-properties-type)`tools[].parameters.properties.type` The type of this parameter. **Type**: `string` ### [](#tools-parameters-required)`tools[].parameters.required[]` The required parameters for this pipeline. **Type**: `array` **Default**: `[]` ### [](#tools-processors)`tools[].processors[]` The pipeline to execute when the LLM uses this tool. **Type**: `processor` ### [](#top_p)`top_p` An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with `top_p` probability mass. For example, a `top_p` of `0.1` means only the tokens comprising the top 10% probability mass are sampled. Redpanda recommends adding a value for this field or `temperature`, but not both. **Type**: `float` ## [](#example)Example In this pipeline configuration, the Command R+ model executes a number of processors, which make a tool call to retrieve weather data for a specific city. ```yaml input: generate: count: 1 mapping: | root = "What is the weather like in Chicago?" pipeline: processors: - cohere_chat: auth_token: my_cohere_api_token model: command-r-plus prompt: "${!content().string()}" tools: - name: GetWeather description: "Retrieve the weather for a specific city" parameters: required: ["city"] properties: city: type: string description: the city to look up the weather for processors: - http: verb: GET url: 'https://wttr.in/${!this.city}?T' headers: User-Agent: curl/8.11.1 # Returns a text string from the weather website output: stdout: {} ``` --- # Page 224: cohere_embeddings **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/cohere_embeddings.md --- # cohere\_embeddings --- title: cohere_embeddings latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/cohere_embeddings page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/cohere_embeddings.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/cohere_embeddings.adoc page-git-created-date: "2024-10-16" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/cohere_embeddings/ "View the Cloud version of this component"), Self-Managed Generates vector embeddings to represent input text, using the [Cohere API](https://docs.cohere.com/docs/embeddings). Introduced in version 4.37.0. ```yml # Configuration fields, showing default values label: "" cohere_embeddings: base_url: https://api.cohere.com auth_token: "" # No default (required) model: embed-english-v3.0 # No default (required) text_mapping: "" # No default (optional) input_type: search_document dimensions: "" # No default (optional) ``` This processor sends text strings to your chosen large language model (LLM), which generates vector embeddings for them using the Cohere API. By default, the processor submits the entire payload of each message as a string, unless you use the `text_mapping` field to customize it. To learn more about vector embeddings, see the [Cohere API documentation](https://docs.cohere.com/docs/embeddings). ## [](#examples)Examples ### [](#store-embedding-vectors-in-qdrant)Store embedding vectors in Qdrant Compute embeddings for some generated data and store it within xrefs:component:outputs/qdrant.adoc\[Qdrant\] ```yaml input: generate: interval: 1s mapping: | root = {"text": fake("paragraph")} pipeline: processors: - cohere_embeddings: model: embed-english-v3 api_key: "${COHERE_API_KEY}" text_mapping: "root = this.text" output: qdrant: grpc_host: localhost:6334 collection_name: "example_collection" id: "root = uuid_v4()" vector_mapping: "root = this" ``` ## [](#fields)Fields ### [](#api_key)`api_key` The API key for the Cohere API. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#base_url)`base_url` The base URL to use for API requests. **Type**: `string` **Default**: `[https://api.cohere.com](https://api.cohere.com)` ### [](#dimensions)`dimensions` The number of dimensions (numerical values) in each vector embedding generated by this processor. This parameter only supports [`embed-v4.0`](https://docs.cohere.com/v2/docs/embeddings) and newer models. **Type**: `int` ### [](#input_type)`input_type` The type of text input passed to the model. **Type**: `string` **Default**: `search_document` | Option | Summary | | --- | --- | | classification | Used for embeddings passed through a text classifier. | | clustering | Used for the embeddings run through a clustering algorithm. | | search_document | Used for embeddings stored in a vector database for search use-cases. | | search_query | Used for embeddings of search queries run against a vector DB to find relevant documents. | ### [](#model)`model` The name of the Cohere LLM you want to use. **Type**: `string` ```yaml # Examples: model: embed-english-v3.0 # --- model: embed-english-light-v3.0 # --- model: embed-multilingual-v3.0 # --- model: embed-multilingual-light-v3.0 ``` ### [](#text_mapping)`text_mapping` The text you want to generate a vector embedding for. By default, the processor submits the entire payload as a string. **Type**: `string` --- # Page 225: cohere_rerank **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/cohere_rerank.md --- # cohere\_rerank --- title: cohere_rerank latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/cohere_rerank page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/cohere_rerank.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/cohere_rerank.adoc page-git-created-date: "2025-05-19" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/cohere_rerank/ "View the Cloud version of this component"), Self-Managed Sends document strings to the [Cohere API](https://docs.cohere.com/reference/rerank), which returns them [ranked by their relevance to a specified query](https://docs.cohere.com/docs/rerank-2). The output of this processor is an array of strings, ordered by their relevance to the query. Introduced in version 4.53.0. ```yml # Configuration fields, showing default values label: "" cohere_rerank: base_url: https://api.cohere.com api_key: "" # No default (required) model: rerank-v3.5 # No default (required) query: "" # No default (required) documents: "" # No default (required) top_n: 0 max_tokens_per_doc: 4096 ``` ## [](#metadata)Metadata - `relevance_scores`: An array of scores for each input document that indicates how relevant it is to the query. The scores are in the same order as the documents in the input. The higher the score, the more relevant the document. ## [](#examples)Examples ### [](#rerank-some-documents-based-on-a-query)Rerank some documents based on a query Rerank some documents based on a query ```yaml input: generate: interval: 1s mapping: | root = { "query": fake("sentence"), "docs": [fake("paragraph"), fake("paragraph"), fake("paragraph")], } pipeline: processors: - cohere_rerank: model: rerank-v3.5 api_key: "${COHERE_API_KEY}" query: "${!this.query}" documents: "root = this.docs" output: stdout: {} ``` ## [](#fields)Fields ### [](#api_key)`api_key` Your API key for the Cohere API. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#base_url)`base_url` The base URL to use for API requests. **Type**: `string` **Default**: `[https://api.cohere.com](https://api.cohere.com)` ### [](#documents)`documents` A list of text strings that are compared to the specified query. For optimal performance: - Send fewer than 1000 documents in a single request - Send structured data in YAML format **Type**: `string` ### [](#max_tokens_per_doc)`max_tokens_per_doc` This processor automatically truncates long documents to the specified number of tokens. **Type**: `int` **Default**: `4096` ### [](#model)`model` The name of the Cohere LLM you want to use. **Type**: `string` ```yaml # Examples: model: rerank-v3.5 ``` ### [](#query)`query` The search query you want to execute. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#top_n)`top_n` The number of documents to return when the query is executed. If set to `0`, all documents are returned. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `0` --- # Page 226: command **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/command.md --- # command --- title: command latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/command page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/command.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/command.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Available in:** Self-Managed Executes a command for each message. Introduced in version 4.21.0. ```yml # Config fields, showing default values label: "" command: name: bash # No default (required) args_mapping: '[ "-c", this.script_path ]' # No default (optional) ``` The specified command is executed for each message processed, with the raw bytes of the message being fed into the stdin of the command process, and the resulting message having its contents replaced with the stdout of it. ## [](#performance)Performance Since this processor executes a new process for each message performance will likely be an issue for high throughput streams. If this is the case then consider using the [`subprocess` processor](../subprocess/) instead as it keeps the underlying process alive long term and uses codecs to insert and extract inputs and outputs to it via stdin/stdout. ## [](#error-handling)Error handling If a non-zero error code is returned by the command then an error containing the entirety of stderr (or a generic message if nothing is written) is set on the message. These failed messages will continue through the pipeline unchanged, but can be dropped or placed in a dead letter queue according to your config, you can read about [these patterns](../../../configuration/error_handling/). If the command is successful but stderr is written to then a metadata field `command_stderr` is populated with its contents. ## [](#fields)Fields ### [](#args_mapping)`args_mapping` An optional [Bloblang mapping](../../../guides/bloblang/about/) that, when specified, should resolve into an array of arguments to pass to the command. Command arguments are expressed this way in order to support dynamic behavior. **Type**: `string` ```yaml # Examples: args_mapping: [ "-c", this.script_path ] ``` ### [](#name)`name` The name of the command to execute. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: name: bash # --- name: go # --- name: ${! @command } ``` ## [](#examples)Examples ### [](#cron-scheduled-command)Cron Scheduled Command This example uses a [`generate` input](../../inputs/generate/) to trigger a command on a cron schedule: ```yaml input: generate: interval: '0,30 */2 * * * *' mapping: 'root = ""' # Empty string as we do not need to pipe anything to stdin processors: - command: name: df args_mapping: '[ "-h" ]' ``` ### [](#dynamic-command-execution)Dynamic Command Execution This example config takes structured messages of the form `{"command":"echo","args":["foo"]}` and uses their contents to execute the contained command and arguments dynamically, replacing its contents with the command result printed to stdout: ```yaml pipeline: processors: - command: name: ${! this.command } args_mapping: 'this.args' ``` --- # Page 227: compress **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/compress.md --- # compress --- title: compress latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/compress page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/compress.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/compress.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/compress/ "View the Cloud version of this component"), Self-Managed Compresses messages according to the selected algorithm. Supported compression algorithms are: \[flate gzip lz4 pgzip snappy zlib\] ```yml # Config fields, showing default values label: "" compress: algorithm: "" # No default (required) level: -1 ``` The 'level' field might not apply to all algorithms. ## [](#fields)Fields ### [](#algorithm)`algorithm` The compression algorithm to use. **Type**: `string` **Options**: `flate`, `gzip`, `lz4`, `pgzip`, `snappy`, `zlib` ### [](#level)`level` The level of compression to use. May not be applicable to all algorithms. **Type**: `int` **Default**: `-1` --- # Page 228: couchbase **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/couchbase.md --- # couchbase --- title: couchbase latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/couchbase page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/couchbase.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/couchbase.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Processor ▼ [Processor](/redpanda-connect/components/processors/couchbase/)[Cache](/redpanda-connect/components/caches/couchbase/)[Output](/redpanda-connect/components/outputs/couchbase/) **Available in:** Self-Managed Performs operations against Couchbase for each message, allowing you to store or retrieve data within message payloads. Introduced in version 4.11.0. #### Common ```yml processors: label: "" couchbase: url: "" # No default (required) username: "" # No default (optional) password: "" # No default (optional) bucket: "" # No default (required) id: "" # No default (required) content: "" # No default (optional) operation: get ``` #### Advanced ```yml processors: label: "" couchbase: url: "" # No default (required) username: "" # No default (optional) password: "" # No default (optional) bucket: "" # No default (required) collection: "" # No default (optional) scope: "" # No default (optional) transcoder: legacy timeout: 15s id: "" # No default (required) content: "" # No default (optional) ttl: "" # No default (optional) operation: get ``` When inserting, replacing or upserting documents, each must have the `content` property set. ## [](#fields)Fields ### [](#bucket)`bucket` Couchbase bucket. **Type**: `string` ### [](#collection)`collection` Bucket collection. **Type**: `string` ### [](#content)`content` Document content. **Type**: `string` ### [](#id)`id` Document id. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: id: ${! json("id") } ``` ### [](#operation)`operation` Couchbase operation to perform. **Type**: `string` **Default**: `get` | Option | Summary | | --- | --- | | get | fetch a document. | | insert | insert a new document. | | remove | delete a document. | | replace | replace the contents of a document. | | upsert | creates a new document if it does not exist, if it does exist then it updates it. | ### [](#password)`password` Password to connect to the cluster. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#scope)`scope` Bucket scope. **Type**: `string` ### [](#timeout)`timeout` Operation timeout. **Type**: `string` **Default**: `15s` ### [](#transcoder)`transcoder` Couchbase transcoder to use. **Type**: `string` **Default**: `legacy` | Option | Summary | | --- | --- | | json | JSONTranscoder implements the default transcoding behavior and applies JSON transcoding to all values. This will apply the following behavior to the value: binary ([]byte) → error. default → JSON value, JSON Flags. | | legacy | LegacyTranscoder implements the behavior for a backward-compatible transcoder. This transcoder implements behavior matching that of gocb v1.This will apply the following behavior to the value: binary ([]byte) → binary bytes, Binary expectedFlags. string → string bytes, String expectedFlags. default → JSON value, JSON expectedFlags. | | raw | RawBinaryTranscoder implements passthrough behavior of raw binary data. This transcoder does not apply any serialization. This will apply the following behavior to the value: binary ([]byte) → binary bytes, binary expectedFlags. default → error. | | rawjson | RawJSONTranscoder implements passthrough behavior of JSON data. This transcoder does not apply any serialization. It will forward data across the network without incurring unnecessary parsing costs. This will apply the following behavior to the value: binary ([]byte) → JSON bytes, JSON expectedFlags. string → JSON bytes, JSON expectedFlags. default → error. | | rawstring | RawStringTranscoder implements passthrough behavior of raw string data. This transcoder does not apply any serialization. This will apply the following behavior to the value: string → string bytes, string expectedFlags. default → error. | ### [](#ttl)`ttl` An optional TTL to set for items. **Type**: `string` ### [](#url)`url` Couchbase connection string. **Type**: `string` ```yaml # Examples: url: couchbase://localhost:11210 ``` ### [](#username)`username` Username to connect to the cluster. **Type**: `string` --- # Page 229: crash **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/crash.md --- # crash --- title: crash latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/crash page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/crash.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/crash.adoc page-git-created-date: "2025-03-11" page-git-modified-date: "2025-03-11" --- Stops the current pipeline process with a fatal log message. You can use [function interpolations](../../../configuration/interpolation/#bloblang-queries) to define the log message, including the message contents and metadata that caused the fatal error. ```yml # Configuration fields, showing default values label: "" crash: "" # No default (required) ``` This processor is useful for detecting unhandled errors during development and testing. For more information about common patterns for trapping errors, see [Error Handling](../../../configuration/error_handling/). ## [](#fields)Fields ### [](#crash)`crash` Define the fatal log message that is written when an unhandled error occurs. This field supports [function interpolations](../../../configuration/interpolation/#bloblang-queries). ```none # Examples crash: "Processor ${!error_source_label()} failed due to: ${!error()}" ``` --- # Page 230: decompress **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/decompress.md --- # decompress --- title: decompress latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/decompress page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/decompress.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/decompress.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Type:** Processor ▼ [Processor](/redpanda-connect/components/processors/decompress/)[Scanner](/redpanda-connect/components/scanners/decompress/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/decompress/ "View the Cloud version of this component"), Self-Managed Decompresses messages according to the selected algorithm. Supported decompression algorithms are: \[bzip2 flate gzip lz4 pgzip snappy zlib\] ```yml # Config fields, showing default values label: "" decompress: algorithm: "" # No default (required) ``` ## [](#fields)Fields ### [](#algorithm)`algorithm` The decompression algorithm to use. **Type**: `string` **Options**: `bzip2`, `flate`, `gzip`, `lz4`, `pgzip`, `snappy`, `zlib` --- # Page 231: dedupe **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/dedupe.md --- # dedupe --- title: dedupe latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/dedupe page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/dedupe.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/dedupe.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/dedupe/ "View the Cloud version of this component"), Self-Managed Deduplicates messages by storing a key value in a cache using the `add` operator. If the key already exists within the cache it is dropped. ```yml # Config fields, showing default values label: "" dedupe: cache: "" # No default (required) key: ${! meta("kafka_key") } # No default (required) drop_on_err: true ``` Caches must be configured as resources, for more information check out the [cache documentation](../../caches/about/). When using this processor with an output target that might fail you should always wrap the output within an indefinite [`retry`](../../outputs/retry/) block. This ensures that during outages your messages aren’t reprocessed after failures, which would result in messages being dropped. ## [](#batch-deduplication)Batch deduplication This processor enacts on individual messages only, in order to perform a deduplication on behalf of a batch (or window) of messages instead use the [`cache` processor](../cache/#examples). ## [](#delivery-guarantees)Delivery guarantees Performing deduplication on a stream using a distributed cache voids any at-least-once guarantees that it previously had. This is because the cache will preserve message signatures even if the message fails to leave the Redpanda Connect pipeline, which would cause message loss in the event of an outage at the output sink followed by a restart of the Redpanda Connect instance (or a server crash, etc). This problem can be mitigated by using an in-memory cache and distributing messages to horizontally scaled Redpanda Connect pipelines partitioned by the deduplication key. However, in situations where at-least-once delivery guarantees are important it is worth avoiding deduplication in favour of implement idempotent behavior at the edge of your stream pipelines. ## [](#fields)Fields ### [](#cache)`cache` The [`cache` resource](../../caches/about/) to target with this processor. **Type**: `string` ### [](#drop_on_err)`drop_on_err` Whether messages should be dropped when the cache returns a general error such as a network issue. **Type**: `bool` **Default**: `true` ### [](#key)`key` An interpolated string yielding the key to deduplicate by for each message. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: key: ${! meta("kafka_key") } # --- key: ${! content().hash("xxhash64") } ``` ## [](#examples)Examples ### [](#deduplicate-based-on-kafka-key)Deduplicate based on Kafka key The following configuration demonstrates a pipeline that deduplicates messages based on the Kafka key. ```yaml pipeline: processors: - dedupe: cache: keycache key: ${! meta("kafka_key") } cache_resources: - label: keycache memory: default_ttl: 60s ``` --- # Page 232: ffi **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/ffi.md --- # ffi --- title: ffi latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/ffi page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/ffi.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/ffi.adoc description: Invoke a function within a shared library as a processing step. page-git-created-date: "2026-01-05" page-git-modified-date: "2026-01-05" --- **Available in:** Self-Managed Invoke a function within a shared library as a processing step. #### Common ```yml processors: label: "" ffi: library_path: "" # No default (required) function_name: "" # No default (required) args_mapping: "" # No default (required) signature: return: type: "" # No default (required) parameters: [] # No default (required) ``` #### Advanced ```yml processors: label: "" ffi: library_path: "" # No default (required) function_name: "" # No default (required) args_mapping: "" # No default (required) signature: return: type: "" # No default (required) parameters: [] # No default (required) ``` The `ffi` processor allows you to dynamically load shared libraries (.so, .dylib, or .dll files) and invoke their functions as a processing step. This enables integration with external C/C++ libraries without requiring custom Go code or rebuilding Redpanda Connect. The processor uses `dlopen` (or platform equivalent) to load the specified library at runtime, looks up the function by name, and invokes it with arguments provided through Bloblang mapping. Function signatures are defined using the `signature` field, specifying parameter and return types. Results are returned as an array, where the first element is the return value (if not void), followed by any output parameters in their declaration order. ## [](#requirements)Requirements Ensure that your environment meets the following requirements: - This component is only available in cgo-enabled builds of Redpanda Connect (not in the Redpanda CLI or Docker image). You can either [download a prebuilt cgo-enabled binary](../../../install/prebuilt-binary/) or [build Redpanda Connect from source with cgo enabled](../../../install/build-from-source/). ## [](#fields)Fields ### [](#args_mapping)`args_mapping` The bloblang expression that returns an array of arguments to pass into the foreign function. **Type**: `string` ```yaml # Examples: args_mapping: root = [42, now().ts_unix_nano(), content()] ``` ### [](#function_name)`function_name` The name of the function to load from the shared library. **Type**: `string` ```yaml # Examples: function_name: MyExternCFunction ``` ### [](#library_path)`library_path` The path to the shared library (.so, .dylib or .dll) file to load dynamically. **Type**: `string` ```yaml # Examples: library_path: libbar.6.so # --- library_path: libfoo.dylib ``` ### [](#signature)`signature` The signature of the function. **Type**: `object` ### [](#signature-parameters)`signature.parameters[]` The parameters of the function. **Type**: `object` ### [](#signature-parameters-out)`signature.parameters[].out` If the parameter provided is an 'out' parameter, meaning if the function mutates the value, and the resulting value should be returned. This is only valid for pointer types. **Type**: `bool` **Default**: `false` ### [](#signature-parameters-type)`signature.parameters[].type` The data type of the parameter. **Type**: `string` | Option | Summary | | --- | --- | | byte* | A pointer to a byte array is provided as an argument. Note this byte array cannot be referenced once the function returns. args_mapping must return a byte array or string type for this argument, and the parameter in C for this should be void*. | | int32 | A 32 bit signed integer is provided as an argument | | int64 | A 64 bit signed integer is provided as an argument | ### [](#signature-return)`signature.return` The configuration for the function’s result. **Type**: `object` ### [](#signature-return-type)`signature.return.type` The data type of function’s return value **Type**: `string` | Option | Summary | | --- | --- | | int32 | A 32 bit signed integer is returned | | int64 | A 64 bit signed integer is returned | | void | The function returns nothing | ## [](#examples)Examples ### [](#call-a-libc-function)Call a libc function This is an example of loading libc.so and calling a function on linux. ```yaml pipeline: processors: - ffi: library_path: libc.6.so function_name: memcmp args_mapping: 'root = ["foo", "bar", 3]' signature: return: type: int32 parameters: - type: byte* - type: byte* - type: int64 ``` --- # Page 233: for_each **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/for_each.md --- # for\_each --- title: for_each latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/for_each page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/for_each.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/for_each.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/for_each/ "View the Cloud version of this component"), Self-Managed A processor that applies a list of child processors to messages of a batch as though they were each a batch of one message. ```yml # Config fields, showing default values label: "" for_each: [] ``` This is useful for forcing batch wide processors such as [`dedupe`](../dedupe/) or interpolations such as the `value` field of the `metadata` processor to execute on individual message parts of a batch instead. Please note that most processors already process per message of a batch, and this processor is not needed in those cases. --- # Page 234: gcp_bigquery_select **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/gcp_bigquery_select.md --- # gcp\_bigquery\_select --- title: gcp_bigquery_select latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/gcp_bigquery_select page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/gcp_bigquery_select.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/gcp_bigquery_select.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Processor ▼ [Processor](/redpanda-connect/components/processors/gcp_bigquery_select/)[Input](/redpanda-connect/components/inputs/gcp_bigquery_select/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/gcp_bigquery_select/ "View the Cloud version of this component"), Self-Managed Executes a `SELECT` query against BigQuery and replaces messages with the rows returned. Introduced in version 3.64.0. ```yml # Config fields, showing default values label: "" gcp_bigquery_select: project: "" # No default (required) credentials_json: "" # No default (optional) table: bigquery-public-data.samples.shakespeare # No default (required) columns: [] # No default (required) where: type = ? and created_at > ? # No default (optional) job_labels: {} args_mapping: root = [ "article", now().ts_format("2006-01-02") ] # No default (optional) prefix: "" # No default (optional) suffix: "" # No default (optional) ``` ## [](#examples)Examples ### [](#word-count)Word count Given a stream of English terms, enrich the messages with the word count from Shakespeare’s public works: ```yaml pipeline: processors: - branch: processors: - gcp_bigquery_select: project: test-project table: bigquery-public-data.samples.shakespeare columns: - word - sum(word_count) as total_count where: word = ? suffix: | GROUP BY word ORDER BY total_count DESC LIMIT 10 args_mapping: root = [ this.term ] result_map: | root.count = this.get("0.total_count") ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` An optional [Bloblang mapping](../../../guides/bloblang/about/) which should evaluate to an array of values matching in size to the number of placeholder arguments in the field `where`. **Type**: `string` ```yaml # Examples: args_mapping: root = [ "article", now().ts_format("2006-01-02") ] ``` ### [](#columns)`columns[]` A list of columns to query. **Type**: `array` ### [](#credentials_json)`credentials_json` Base64-encoded Google Service Account credentials in JSON format (optional). Use this field to authenticate with Google Cloud services. For more information about creating service account credentials, see [Google’s service account documentation](https://developers.google.com/workspace/guides/create-credentials#create_credentials_for_a_service_account). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#job_labels)`job_labels` A list of labels to add to the query job. **Type**: `string` **Default**: `{}` ### [](#prefix)`prefix` An optional prefix to prepend to the select query (before SELECT). **Type**: `string` ### [](#project)`project` GCP project where the query job will execute. **Type**: `string` ### [](#suffix)`suffix` An optional suffix to append to the select query. **Type**: `string` ### [](#table)`table` Fully-qualified BigQuery table name to query. **Type**: `string` ```yaml # Examples: table: bigquery-public-data.samples.shakespeare ``` ### [](#where)`where` An optional where clause to add. Placeholder arguments are populated with the `args_mapping` field. Placeholders should always be question marks (`?`). **Type**: `string` ```yaml # Examples: where: type = ? and created_at > ? # --- where: user_id = ? ``` --- # Page 235: gcp_vertex_ai_chat **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/gcp_vertex_ai_chat.md --- # gcp\_vertex\_ai\_chat --- title: gcp_vertex_ai_chat latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/gcp_vertex_ai_chat page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/gcp_vertex_ai_chat.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/gcp_vertex_ai_chat.adoc page-git-created-date: "2024-09-04" page-git-modified-date: "2026-01-13" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/gcp_vertex_ai_chat/ "View the Cloud version of this component"), Self-Managed Generates responses to messages in a chat conversation, using the [Vertex API AI](https://cloud.google.com/vertex-ai/docs/start/introduction-unified-platform). Introduced in version 4.34.0. #### Common ```yml processors: label: "" gcp_vertex_ai_chat: project: "" # No default (required) credentials_json: "" # No default (optional) location: "" # No default (required) model: "" # No default (required) prompt: "" # No default (optional) history: "" # No default (optional) attachment: "" # No default (optional) temperature: "" # No default (optional) max_tokens: "" # No default (optional) response_format: text tools: [] ``` #### Advanced ```yml processors: label: "" gcp_vertex_ai_chat: project: "" # No default (required) credentials_json: "" # No default (optional) location: "" # No default (required) model: "" # No default (required) prompt: "" # No default (optional) system_prompt: "" # No default (optional) history: "" # No default (optional) attachment: "" # No default (optional) temperature: "" # No default (optional) max_tokens: "" # No default (optional) response_format: text top_p: "" # No default (optional) top_k: "" # No default (optional) stop: [] # No default (optional) presence_penalty: "" # No default (optional) frequency_penalty: "" # No default (optional) max_tool_calls: 10 tools: [] ``` This processor sends prompts to your chosen large language model (LLM) and generates text from the responses, using the Vertex AI API. For more information, see the [Vertex AI documentation](https://cloud.google.com/vertex-ai/docs). ## [](#fields)Fields ### [](#attachment)`attachment` Additional data like an image to send with the prompt to the model. The result of the mapping must be a byte array, and the content type is automatically detected. Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: attachment: root = this.image.decode("base64") # decode base64 encoded image ``` ### [](#credentials_json)`credentials_json` An optional field to set a Google Service Account Credentials JSON. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#frequency_penalty)`frequency_penalty` Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model’s likelihood to repeat the same line verbatim. **Type**: `float` ### [](#history)`history` Historical messages to include in the chat request. The result of the bloblang query should be an array of objects of the form of \[{"role": "", "content":""}\], where role is "user" or "model". **Type**: `string` ### [](#location)`location` Specify the location of a fine tuned model. For base models, you can omit this field. **Type**: `string` ```yaml # Examples: location: us-central1 ``` ### [](#max_tokens)`max_tokens` The maximum number of output tokens to generate per message. **Type**: `int` ### [](#max_tool_calls)`max_tool_calls` The maximum number of sequential tool calls. **Type**: `int` **Default**: `10` ### [](#model)`model` The name of the LLM to use. For a full list of models, see the [Vertex AI Model Garden](https://console.cloud.google.com/vertex-ai/model-garden). **Type**: `string` ```yaml # Examples: model: gemini-1.5-pro-001 # --- model: gemini-1.5-flash-001 ``` ### [](#presence_penalty)`presence_penalty` Positive values penalize new tokens if they appear in the text already, increasing the model’s likelihood to include new topics. **Type**: `float` ### [](#project)`project` The GCP project ID to use. **Type**: `string` ### [](#prompt)`prompt` The prompt you want to generate a response for. By default, the processor submits the entire payload as a string. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#response_format)`response_format` The format of the generated response. You must also prompt the model to output the appropriate response type. **Type**: `string` **Default**: `text` **Options**: `text`, `json` ### [](#stop)`stop[]` Sets the stop sequences to use. When this pattern is encountered the LLM stops generating text and returns the final response. **Type**: `array` ### [](#system_prompt)`system_prompt` The system prompt to submit to the Vertex AI LLM. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#temperature)`temperature` Controls the randomness of predictions. **Type**: `float` ### [](#tools)`tools[]` The tools to allow the LLM to invoke. This allows building subpipelines that the LLM can choose to invoke to execute agentic-like actions. **Type**: `object` **Default**: `[]` ### [](#tools-description)`tools[].description` A description of this tool, the LLM uses this to decide if the tool should be used. **Type**: `string` ### [](#tools-name)`tools[].name` The name of this tool. **Type**: `string` ### [](#tools-parameters)`tools[].parameters` The parameters the LLM needs to provide to invoke this tool. **Type**: `object` ### [](#tools-parameters-properties)`tools[].parameters.properties` The properties for the processor’s input data **Type**: `object` ### [](#tools-parameters-properties-description)`tools[].parameters.properties.description` A description of this parameter. **Type**: `string` ### [](#tools-parameters-properties-enum)`tools[].parameters.properties.enum[]` Specifies that this parameter is an enum and only these specific values should be used. **Type**: `array` **Default**: `[]` ### [](#tools-parameters-properties-type)`tools[].parameters.properties.type` The type of this parameter. **Type**: `string` ### [](#tools-parameters-required)`tools[].parameters.required[]` The required parameters for this pipeline. **Type**: `array` **Default**: `[]` ### [](#tools-processors)`tools[].processors[]` The pipeline to execute when the LLM uses this tool. **Type**: `processor` ### [](#top_k)`top_k` Enables top-k sampling (optional). **Type**: `float` ### [](#top_p)`top_p` Enables nucleus sampling (optional). **Type**: `float` --- # Page 236: gcp_vertex_ai_embeddings **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/gcp_vertex_ai_embeddings.md --- # gcp\_vertex\_ai\_embeddings --- title: gcp_vertex_ai_embeddings latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/gcp_vertex_ai_embeddings page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/gcp_vertex_ai_embeddings.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/gcp_vertex_ai_embeddings.adoc page-git-created-date: "2024-10-16" page-git-modified-date: "2026-01-13" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/gcp_vertex_ai_embeddings/ "View the Cloud version of this component"), Self-Managed Generates vector embeddings to represent a text string, using the [Vertex AI API](https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings). Introduced in version 4.37.0. ```yml # Configuration fields, showing default values label: "" gcp_vertex_ai_embeddings: project: "" # No default (required) credentials_json: "" # No default (optional) location: us-central1 model: text-embedding-004 # No default (required) task_type: RETRIEVAL_DOCUMENT text: "" # No default (optional) output_dimensions: 0 # No default (optional) ``` This processor sends text strings to the Vertex AI API, which generates vector embeddings for them. By default, the processor submits the entire payload of each message as a string, unless you use the `text` field to customize it. For more information, see the [Vertex AI documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings). ## [](#fields)Fields ### [](#credentials_json)`credentials_json` Set your Google Service Account Credentials as JSON. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#location)`location` The location of the Vertex AI large language model (LLM) that you want to use. **Type**: `string` **Default**: `us-central1` ### [](#model)`model` The name of the LLM to use. For a full list of models, see the [Vertex AI Model Garden](https://console.cloud.google.com/vertex-ai/model-garden). **Type**: `string` ```yaml # Examples: model: text-embedding-004 # --- model: text-multilingual-embedding-002 ``` ### [](#output_dimensions)`output_dimensions` The maximum length of a generated vector embedding. If this value is set, generated embeddings are truncated to this size. **Type**: `int` ### [](#project)`project` The ID of your Google Cloud project. **Type**: `string` ### [](#task_type)`task_type` Use the following options to optimize embeddings that the model generates for specific use cases. **Type**: `string` **Default**: `RETRIEVAL_DOCUMENT` | Option | Summary | | --- | --- | | CLASSIFICATION | optimize for being able classify texts according to preset labels | | CLUSTERING | optimize for clustering texts based on their similarities | | FACT_VERIFICATION | optimize for queries that are proving or disproving a fact such as "apples grow underground" | | QUESTION_ANSWERING | optimize for search proper questions such as "Why is the sky blue?" | | RETRIEVAL_DOCUMENT | optimize for documents that will be searched (also known as a corpus) | | RETRIEVAL_QUERY | optimize for queries such as "What is the best fish recipe?" or "best restaurant in Chicago" | | SEMANTIC_SIMILARITY | optimize for text similarity | ### [](#text)`text` The text you want to generate vector embeddings for. By default, the processor submits the entire payload as a string. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 237: google_drive_download **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/google_drive_download.md --- # google\_drive\_download --- title: google_drive_download latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/google_drive_download page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/google_drive_download.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/google_drive_download.adoc page-git-created-date: "2025-05-19" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/google_drive_download/ "View the Cloud version of this component"), Self-Managed **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Downloads files from Google Drive that contain matching file IDs. Try out the [example pipeline on this page](#example), which downloads all files from your Google Drive. Introduced in version 4.53.0. #### Common ```yml processors: label: "" google_drive_download: credentials_json: "" # No default (optional) file_id: "" # No default (required) mime_type: "" # No default (required) shared_drives: false ``` #### Advanced ```yml processors: label: "" google_drive_download: credentials_json: "" # No default (optional) file_id: "" # No default (required) mime_type: "" # No default (required) export_mime_types: application/vnd.google-apps.document: "text/markdown" application/vnd.google-apps.drawing: "image/png" application/vnd.google-apps.presentation: "application/pdf" application/vnd.google-apps.script: "application/vnd.google-apps.script+json" application/vnd.google-apps.spreadsheet: "text/csv" shared_drives: false ``` ## [](#authentication)Authentication By default, this processor uses [Google Application Default Credentials (ADC)](https://cloud.google.com/docs/authentication/application-default-credentials) to authenticate with Google APIs. To set up local ADC authentication, use the following `gcloud` commands: - Authenticate using Application Default Credentials and grant read-only access to your Google Drive. ```bash gcloud auth application-default login --scopes='openid,https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/drive.readonly' ``` - Assign a quota project to the Application Default Credentials when using a user account. ```bash gcloud auth application-default set-quota-project ``` Replace the `` placeholder with your Google Cloud project ID To use a service account instead, create a JSON key for the account and add it to the [`credentials_json`](#credentials_json) field. To access Google Drive files using a service account, either: - Explicitly share files with the service account’s email account - Use [domain-wide delegation](https://support.google.com/a/answer/162106) to share all files within a Google Workspace ## [](#fields)Fields ### [](#credentials_json)`credentials_json` The JSON key for your service account (optional). If left empty, Application Default Credentials are used. For more details, see [Authentication](#authentication). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#export_mime_types)`export_mime_types` Maps Google Drive MIME types to [supported file export formats](https://developers.google.com/workspace/drive/api/guides/ref-export-formats). The MIME type is the key, and the export format is the value. **Type**: `string` **Default**: ```yaml application/vnd.google-apps.document: "text/markdown" application/vnd.google-apps.drawing: "image/png" application/vnd.google-apps.presentation: "application/pdf" application/vnd.google-apps.script: "application/vnd.google-apps.script+json" application/vnd.google-apps.spreadsheet: "text/csv" ``` ```yaml # Examples: export_mime_types: application/vnd.google-apps.document: application/pdf application/vnd.google-apps.drawing: application/pdf application/vnd.google-apps.presentation: application/pdf application/vnd.google-apps.spreadsheet: application/pdf # --- export_mime_types: application/vnd.google-apps.document: application/vnd.openxmlformats-officedocument.wordprocessingml.document application/vnd.google-apps.drawing: image/svg+xml application/vnd.google-apps.presentation: application/vnd.openxmlformats-officedocument.presentationml.presentation application/vnd.google-apps.spreadsheet: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet ``` ### [](#file_id)`file_id` The ID of the file to download from Google Drive. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#mime_type)`mime_type` The [MIME type](https://developers.google.com/workspace/drive/api/guides/mime-types) of the file for download. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#shared_drives)`shared_drives` Whether or not to include shared drives. **Type**: `bool` **Default**: `false` ## [](#example)Example This example downloads all files from a Google Drive. ```yaml input: stdin: {} pipeline: processors: - google_drive_search: query: "${!content().string()}" - mutation: 'meta path = this.name' - google_drive_download: file_id: "${!this.id}" mime_type: "${!this.mimeType}" output: file: path: "${!@path}" codec: all-bytes ``` --- # Page 238: google_drive_list_labels **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/google_drive_list_labels.md --- # google\_drive\_list\_labels --- title: google_drive_list_labels latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/google_drive_list_labels page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/google_drive_list_labels.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/google_drive_list_labels.adoc page-git-created-date: "2025-05-19" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/google_drive_list_labels/ "View the Cloud version of this component"), Self-Managed **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Lists [labels](https://developers.google.com/workspace/drive/api/guides/about-labels) for files on a Google Drive. Introduced in version 4.53.0. ```yml # Configuration fields, showing default values label: "" google_drive_list_labels: credentials_json: "" # No default (optional) ``` ## [](#authentication)Authentication By default, this processor uses [Google Application Default Credentials (ADC)](https://cloud.google.com/docs/authentication/application-default-credentials) to authenticate with Google APIs. To set up local ADC authentication, use the following `gcloud` commands: - Authenticate using Application Default Credentials and grant read-only access to your Google Drive. ```bash gcloud auth application-default login --scopes='openid,https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/drive.readonly' ``` - Assign a quota project to the Application Default Credentials when using a user account. ```bash gcloud auth application-default set-quota-project ``` Replace the `` placeholder with your Google Cloud project ID To use a service account instead, create a JSON key for the account and add it to the [`credentials_json`](#credentials_json) field. To access Google Drive files using a service account, either: - Explicitly share files with the service account’s email account - Use [domain-wide delegation](https://support.google.com/a/answer/162106) to share all files within a Google Workspace ## [](#fields)Fields ### [](#credentials_json)`credentials_json` The JSON key for your service account (optional). If left empty, Application Default Credentials are used. For more details, see [Authentication](#authentication). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` --- # Page 239: google_drive_search **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/google_drive_search.md --- # google\_drive\_search --- title: google_drive_search latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/google_drive_search page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/google_drive_search.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/google_drive_search.adoc page-git-created-date: "2025-05-19" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/google_drive_search/ "View the Cloud version of this component"), Self-Managed **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Searches Google Drive for files that match a specified query and emits the results as a batch of messages. Each message contains the [metadata of a Google Drive file](https://developers.google.com/workspace/drive/api/reference/rest/v3/files#File). Try out the [example pipeline on this page](#example), which searches for and downloads all Google Drive files that match the specified query. Introduced in version 4.53.0. ```yml # Configuration fields, showing default values label: "" google_drive_search: credentials_json: "" # No default (optional) query: "" # No default (required) projection: - id - name - mimeType - size - labelInfo include_label_ids: "" # No default (optional) max_results: 64 ``` ## [](#authentication)Authentication By default, this processor uses [Google Application Default Credentials (ADC)](https://cloud.google.com/docs/authentication/application-default-credentials) to authenticate with Google APIs. To set up local ADC authentication, use the following `gcloud` commands: - Authenticate using Application Default Credentials and grant read-only access to your Google Drive. ```bash gcloud auth application-default login --scopes='openid,https://www.googleapis.com/auth/userinfo.email,https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/drive.readonly' ``` - Assign a quota project to the Application Default Credentials when using a user account. ```bash gcloud auth application-default set-quota-project ``` Replace the `` placeholder with your Google Cloud project ID To use a service account instead, create a JSON key for the account and add it to the [`credentials_json`](#credentials_json) field. To access Google Drive files using a service account, either: - Explicitly share files with the service account’s email account - Use [domain-wide delegation](https://support.google.com/a/answer/162106) to share all files within a Google Workspace ## [](#fields)Fields ### [](#credentials_json)`credentials_json` The JSON key for your service account (optional). If left empty, Application Default Credentials are used. For more details, see [Authentication](#authentication). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#include_label_ids)`include_label_ids` A comma delimited list of label IDs to include in the Google Drive search result. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#max_results)`max_results` The maximum number of search results to return. **Type**: `int` **Default**: `64` ### [](#projection)`projection[]` Partial fields to include in the Google Drive search result. **Type**: `array` **Default**: ```yaml - "id" - "name" - "mimeType" - "size" - "labelInfo" ``` ### [](#query)`query` Specify a search query to locate matching files in Google Drive. This field supports: - The same query syntax as the Google Drive UI - [Bloblang interpolation functions](../../../configuration/interpolation/#bloblang-queries) for dynamic query generation **Type**: `string` ### [](#shared_drives)`shared_drives` Whether or not to include shared drives in the result. **Type**: `bool` **Default**: `false` ## [](#example)Example This example searches Google Drive for files matching a query and downloads each file to a specified location. It uses the `google_drive_search` processor to perform the search and the [`google_drive_download` processor](../google_drive_download/) to retrieve the files. ```yaml input: stdin: {} pipeline: processors: - google_drive_search: query: "${!content().string()}" - mutation: 'meta path = this.name' - google_drive_download: file_id: "${!this.id}" mime_type: "${!this.mimeType}" output: file: path: "${!@path}" codec: all-bytes ``` --- # Page 240: grok **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/grok.md --- # grok --- title: grok latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/grok page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/grok.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/grok.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Available in:** Self-Managed Parses messages into a structured format by attempting to apply a list of Grok expressions, the first expression to result in at least one value replaces the original message with a JSON object containing the values. #### Common ```yml processors: label: "" grok: expressions: [] # No default (required) pattern_definitions: {} pattern_paths: [] ``` #### Advanced ```yml processors: label: "" grok: expressions: [] # No default (required) pattern_definitions: {} pattern_paths: [] named_captures_only: true use_default_patterns: true remove_empty_values: true ``` Type hints within patterns are respected, therefore with the pattern `%\{WORD:first},%{INT:second:int}` and a payload of `foo,1` the resulting payload would be `\{"first":"foo","second":1}`. ## [](#performance)Performance This processor currently uses the [Go RE2](https://golang.org/s/re2syntax) regular expression engine, which is guaranteed to run in time linear to the size of the input. However, this property often makes it less performant than PCRE based implementations of grok. For more information, see [https://swtch.com/~rsc/regexp/regexp1.html](https://swtch.com/~rsc/regexp/regexp1.html). ## [](#examples)Examples ### [](#vpc-flow-logs)VPC Flow Logs Grok can be used to parse unstructured logs such as VPC flow logs that look like this: ```text 2 123456789010 eni-1235b8ca123456789 172.31.16.139 172.31.16.21 20641 22 6 20 4249 1418530010 1418530070 ACCEPT OK ``` Into structured objects that look like this: ```json {"accountid":"123456789010","action":"ACCEPT","bytes":4249,"dstaddr":"172.31.16.21","dstport":22,"end":1418530070,"interfaceid":"eni-1235b8ca123456789","logstatus":"OK","packets":20,"protocol":6,"srcaddr":"172.31.16.139","srcport":20641,"start":1418530010,"version":2} ``` With the following config: ```yaml pipeline: processors: - grok: expressions: - '%{VPCFLOWLOG}' pattern_definitions: VPCFLOWLOG: '%{NUMBER:version:int} %{NUMBER:accountid} %{NOTSPACE:interfaceid} %{NOTSPACE:srcaddr} %{NOTSPACE:dstaddr} %{NOTSPACE:srcport:int} %{NOTSPACE:dstport:int} %{NOTSPACE:protocol:int} %{NOTSPACE:packets:int} %{NOTSPACE:bytes:int} %{NUMBER:start:int} %{NUMBER:end:int} %{NOTSPACE:action} %{NOTSPACE:logstatus}' ``` ## [](#fields)Fields ### [](#expressions)`expressions[]` One or more Grok expressions to attempt against incoming messages. The first expression to match at least one value will be used to form a result. **Type**: `array` ### [](#named_captures_only)`named_captures_only` Whether to only capture values from named patterns. **Type**: `bool` **Default**: `true` ### [](#pattern_definitions)`pattern_definitions` A map of pattern definitions that can be referenced within `patterns`. **Type**: `string` **Default**: `{}` ### [](#pattern_paths)`pattern_paths[]` A list of paths to load Grok patterns from. This field supports wildcards, including super globs (double star). **Type**: `array` **Default**: `[]` ### [](#remove_empty_values)`remove_empty_values` Whether to remove values that are empty from the resulting structure. **Type**: `bool` **Default**: `true` ### [](#use_default_patterns)`use_default_patterns` Whether to use a [default set of patterns](#default-patterns). **Type**: `bool` **Default**: `true` ## [](#default-patterns)Default patterns For summary of the default patterns on offer, see [https://github.com/Jeffail/grok/blob/master/patterns.go#L5](https://github.com/Jeffail/grok/blob/master/patterns.go#L5). --- # Page 241: group_by_value **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/group_by_value.md --- # group\_by\_value --- title: group_by_value latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/group_by_value page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/group_by_value.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/group_by_value.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/group_by_value/ "View the Cloud version of this component"), Self-Managed Splits a batch of messages into N batches, where each resulting batch contains a group of messages determined by a [function interpolated string](../../../configuration/interpolation/#bloblang-queries) evaluated per message. ```yml # Config fields, showing default values label: "" group_by_value: value: ${! meta("kafka_key") } # No default (required) ``` This allows you to group messages using arbitrary fields within their content or metadata, process them individually, and send them to unique locations as per their group. The functionality of this processor depends on being applied across messages that are batched. You can find out more about batching [in this doc](../../../configuration/batching/). ## [](#fields)Fields ### [](#value)`value` The interpolated string to group based on. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: value: ${! meta("kafka_key") } # --- value: ${! json("foo.bar") }-${! meta("baz") } ``` ## [](#examples)Examples If we were consuming Kafka messages and needed to group them by their key, archive the groups, and send them to S3 with the key as part of the path we could achieve that with the following: ```yaml pipeline: processors: - group_by_value: value: ${! meta("kafka_key") } - archive: format: tar - compress: algorithm: gzip output: aws_s3: bucket: TODO path: docs/${! meta("kafka_key") }/${! count("files") }-${! timestamp_unix_nano() }.tar.gz ``` --- # Page 242: group_by **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/group_by.md --- # group\_by --- title: group_by latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/group_by page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/group_by.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/group_by.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/group_by/ "View the Cloud version of this component"), Self-Managed Splits a [batch of messages](../../../configuration/batching/) into N batches, where each resulting batch contains a group of messages determined by a [Bloblang query](../../../guides/bloblang/about/). ```yml # Config fields, showing default values label: "" group_by: [] # No default (required) ``` Once the groups are established a list of processors are applied to their respective grouped batch, which can be used to label the batch as per their grouping. Messages that do not pass the check of any specified group are placed in their own group. The functionality of this processor depends on being applied across messages that are batched. You can find out more about batching [in this doc](../../../configuration/batching/). ## [](#fields)Fields ### [](#check)`check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message belongs to a given group. **Type**: `string` ```yaml # Examples: check: this.type == "foo" # --- check: this.contents.urls.contains("https://benthos.dev/") # --- check: true ``` ### [](#processors)`processors[]` A list of [processors](../about/) to execute on the newly formed group. **Type**: `processor` **Default**: `[]` ## [](#examples)Examples ### [](#grouped-processing)Grouped Processing Imagine we have a batch of messages that we wish to split into a group of foos and everything else, which should be sent to different output destinations based on those groupings. We also need to send the foos as a tar gzip archive. For this purpose we can use the `group_by` processor with a [`switch`](../../outputs/switch/) output: ```yaml pipeline: processors: - group_by: - check: content().contains("this is a foo") processors: - archive: format: tar - compress: algorithm: gzip - mapping: 'meta grouping = "foo"' output: switch: cases: - check: meta("grouping") == "foo" output: gcp_pubsub: project: foo_prod topic: only_the_foos - output: gcp_pubsub: project: somewhere_else topic: no_foos_here ``` --- # Page 243: http **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/http.md --- # http --- title: http latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/http page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/http.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/http.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/http/ "View the Cloud version of this component"), Self-Managed Performs a HTTP request using a message batch as the request body, and replaces the original message parts with the body of the response. #### Common ```yml processors: label: "" http: url: "" # No default (required) verb: POST headers: {} rate_limit: "" # No default (optional) timeout: 5s parallel: false ``` #### Advanced ```yml processors: label: "" http: url: "" # No default (required) verb: POST headers: {} metadata: include_prefixes: [] include_patterns: [] dump_request_log_level: "" oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" oauth2: enabled: false client_key: "" client_secret: "" token_url: "" scopes: [] endpoint_params: {} basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] extract_headers: include_prefixes: [] include_patterns: [] rate_limit: "" # No default (optional) timeout: 5s retry_period: 1s max_retry_backoff: 300s retries: 3 follow_redirects: true backoff_on: - 429 drop_on: [] successful_on: [] proxy_url: "" # No default (optional) disable_http2: false batch_as_multipart: false parallel: false ``` ## [](#rate-limit-requests)Rate limit requests You can use the `rate_limit` field to specify a [rate limit resource](../../rate_limits/about/), which restricts the number of requests processed service-wide, regardless of how many components you run in parallel. ## [](#dynamic-url-and-header-settings)Dynamic URL and header settings You can set the [`url`](#url) and [`headers`](#headers) values dynamically using [function interpolations](../../../configuration/interpolation/#bloblang-queries). ## [](#map-payloads-with-the-branch-processor)Map payloads with the branch processor You can use the [`branch` processor](../branch/) to transform or encode the payload into a specific request body format, and map the response back into the original payload instead of replacing it entirely. This example uses a [`branch` processor](../branch/) to strip the request message into an empty body (`request_map: 'root = ""'`), grab an HTTP payload, and place the result back into the original message at the path `repo.status`: ```yaml pipeline: processors: - branch: request_map: 'root = ""' processors: - http: url: https://hub.docker.com/v2/repositories/jeffail/benthos verb: GET headers: Content-Type: application/json result_map: 'root.repo.status = this' ``` ## [](#response-codes)Response codes HTTP response codes in the 200-299 range indicate a successful response. You can use the [`successful_on`](#successful_on) field to add more success status codes. HTTP status codes in the 300-399 range are redirects. The [`follow_redirects` field](#follow_redirects) determines how these responses are handled. If a request returns a response code that matches an entry in: - The [`backoff_on` field](#backoff_on), the request is retried after increasing intervals. - The [`drop_on` field](#drop_on), the request is immediately treated as a failure. ## [](#add-metadata-to-errors)Add metadata to errors If a request returns an error response code, this processor sets a `http_status_code` metadata field in the resulting message. > 💡 **TIP** > > You can use the [`extract_headers`](#extract_headers) field to define rules for copying headers into messages generated from the response. ## [](#error-handling)Error handling When all retry attempts for a message are exhausted, this processor cancels the attempt. By default, the failed message continues through the pipeline unchanged unless you configure other error-handling. For example, you might want to drop failed messages or route them to a dead letter queue. For more information, see [Error Handling](../../../configuration/error_handling/). ## [](#fields)Fields ### [](#backoff_on)`backoff_on[]` A list of status codes that indicate a request failure, and trigger retries with an increasing backoff period between attempts. **Type**: `int` **Default**: ```yaml - 429 ``` ### [](#basic_auth)`basic_auth` Allows you to specify basic authentication. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#batch_as_multipart)`batch_as_multipart` When set to `true`, sends all message in a batch as a single request using [RFC1341](https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html). When set to `false`, sends messages in a batch as individual requests. **Type**: `bool` **Default**: `false` ### [](#disable_http2)`disable_http2` Whether to disable HTTP/2. By default, HTTP/2 is enabled. Requires version 4.44.0 or later. **Type**: `bool` **Default**: `false` ### [](#drop_on)`drop_on[]` A list of status codes that indicate a request failure, where the input should not attempt retries. This helps avoid unnecessary retries for requests that are unlikely to succeed. > 📝 **NOTE** > > In these cases, the _request_ is dropped, but the _message_ that triggered the request is retained. **Type**: `int` **Default**: `[]` ### [](#dump_request_log_level)`dump_request_log_level` EXPERIMENTAL: Set the logging level for the request and response payloads of each HTTP request. Requires version 4.12.0 or later. **Type**: `string` **Default**: `""` **Options**: `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, `FATAL`, \`\` ### [](#extract_headers)`extract_headers` Specify which response headers to add to the resulting messages as metadata. Header keys are automatically converted to lowercase before matching, so make sure that your patterns target the lowercase versions of the expected header keys. **Type**: `object` ### [](#extract_headers-include_patterns)`extract_headers.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#extract_headers-include_prefixes)`extract_headers.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#follow_redirects)`follow_redirects` Whether to follow redirects, including all responses with HTTP status codes in the 300-399 range. If set to `false`, the response message includes only the body, status, and headers from the redirect response, and this processor does not make a request to the URL specified in the `Location` header. **Type**: `bool` **Default**: `true` ### [](#headers)`headers` A map of headers to add to the request. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `{}` ```yaml # Examples: headers: Content-Type: application/octet-stream traceparent: ${! tracing_span().traceparent } ``` ### [](#jwt)`jwt` Beta Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWT tokens provide secure, stateless authentication between services. **Type**: `object` ### [](#jwt-claims)`jwt.claims` A value used to identify the claims that issued the JWT. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` The cryptographic algorithm used to sign the JWT token. Supported algorithms include RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#max_retry_backoff)`max_retry_backoff` The maximum period to wait between failed requests. **Type**: `string` **Default**: `300s` ### [](#metadata)`metadata` Specify matching rules that determine which metadata keys should be added to the HTTP request as headers. **Type**: `object` ### [](#metadata-include_patterns)`metadata.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#metadata-include_prefixes)`metadata.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#oauth)`oauth` Configure OAuth version 1.0 authentication for secure API access. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` The value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` The secret that establishes ownership of the `oauth.access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` A value used to identify the client to the service provider. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` A secret used to establish ownership of the consumer key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#oauth2)`oauth2` Allows you to specify open authentication using OAuth version 2 and the client credentials token flow. **Type**: `object` ### [](#oauth2-client_key)`oauth2.client_key` A value used to identify the client to the token provider. **Type**: `string` **Default**: `""` ### [](#oauth2-client_secret)`oauth2.client_secret` The secret used to establish ownership of the client key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth2-enabled)`oauth2.enabled` Whether to use OAuth version 2 in requests. **Type**: `bool` **Default**: `false` ### [](#oauth2-endpoint_params)`oauth2.endpoint_params` A list of endpoint parameters specified as arrays of strings (optional). Requires version 4.21.0 or later. **Type**: `object` **Default**: `{}` ```yaml # Examples: endpoint_params: bar: - woof foo: - meow - quack ``` ### [](#oauth2-scopes)`oauth2.scopes[]` A list of requested permissions (optional). Requires version 3.45.0 or later. **Type**: `array` **Default**: `[]` ### [](#oauth2-token_url)`oauth2.token_url` The URL of the token provider. **Type**: `string` **Default**: `""` ### [](#parallel)`parallel` When processing batched messages, this field determines whether messages in the batch are sent in parallel. If set to `false`, messages are sent serially. **Type**: `bool` **Default**: `false` ### [](#proxy_url)`proxy_url` A HTTP proxy URL (optional). **Type**: `string` ### [](#rate_limit)`rate_limit` A [rate limit](../../rate_limits/about/) to throttle requests by (optional). **Type**: `string` ### [](#retries)`retries` The maximum number of retry attempts to make. **Type**: `int` **Default**: `3` ### [](#retry_period)`retry_period` The initial period to wait between failed requests before retrying. **Type**: `string` **Default**: `1s` ### [](#successful_on)`successful_on[]` A list of HTTP status codes that should be considered as successful, even if they are not 2XX codes. This is useful for handling cases where non-2XX codes indicate that the request was processed successfully, such as `303 See Other` or `409 Conflict`. By default, all 2XX codes are considered successful unless they are specified in `backoff_on` or `drop_on` fields. **Type**: `int` **Default**: `[]` ### [](#timeout)`timeout` A static timeout to apply to requests. **Type**: `string` **Default**: `5s` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL to connect to. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#verb)`verb` A verb to connect with. **Type**: `string` **Default**: `POST` ```yaml # Examples: verb: POST # --- verb: GET # --- verb: DELETE ``` --- # Page 244: insert_part **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/insert_part.md --- # insert\_part --- title: insert_part latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/insert_part page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/insert_part.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/insert_part.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/insert_part/ "View the Cloud version of this component"), Self-Managed Insert a new message into a batch at an index. If the specified index is greater than the length of the existing batch it will be appended to the end. ```yml # Config fields, showing default values label: "" insert_part: index: -1 content: "" ``` The index can be negative, and if so the message will be inserted from the end counting backwards starting from -1. E.g. if index = -1 then the new message will become the last of the batch, if index = -2 then the new message will be inserted before the last message, and so on. If the negative index is greater than the length of the existing batch it will be inserted at the beginning. The new message will have metadata copied from the first pre-existing message of the batch. This processor will interpolate functions within the 'content' field, you can find a list of functions [here](../../../configuration/interpolation/#bloblang-queries). ## [](#fields)Fields ### [](#content)`content` The content of the message being inserted. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ### [](#index)`index` The index within the batch to insert the message at. **Type**: `int` **Default**: `-1` --- # Page 245: javascript **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/javascript.md --- # javascript --- title: javascript latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/javascript page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/javascript.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/javascript.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Available in:** Self-Managed Executes a JavaScript code block or file for each message. Introduced in version 4.14.0. ```yml # Configuration fields, showing default values label: "" javascript: code: "" # No default (optional) file: "" # No default (optional) global_folders: [] ``` The [execution engine](https://github.com/dop251/goja) behind this processor provides full ECMAScript 5.1 support (including regex and strict mode). Most of the ECMAScript 6 spec is implemented but this is a work in progress. Imports via `require` should work similarly to NodeJS, and access to the console is supported which will print via the Redpanda Connect logger. More caveats can be found on [GitHub](https://github.com/dop251/goja#known-incompatibilities-and-caveats). This processor is implemented using the [github.com/dop251/goja](https://github.com/dop251/goja) library. ## [](#fields)Fields ### [](#code)`code` An inline JavaScript program to run. You must specify a value for either the `code` or `file` field. **Type**: `string` ### [](#file)`file` A file containing a JavaScript program to run. You must specify a value for either the `code` or `file` field. **Type**: `string` ### [](#global_folders)`global_folders[]` A list of directories to load modules from if the requested JavaScript module is not found elsewhere. **Type**: `array` **Default**: `[]` ## [](#examples)Examples ### [](#simple-mutation)Simple mutation In this example we define a simple function that performs a basic mutation against messages, treating their contents as raw strings. ```yaml pipeline: processors: - javascript: code: 'benthos.v0_msg_set_string(benthos.v0_msg_as_string() + "hello world");' ``` ### [](#structured-mutation)Structured mutation In this example we define a function that performs basic mutations against a structured message. Note that we encapsulate the logic within an anonymous function that is called for each invocation, this is required in order to avoid duplicate variable declarations in the global state. ```yaml pipeline: processors: - javascript: code: | (() => { let thing = benthos.v0_msg_as_structured(); thing.num_keys = Object.keys(thing).length; delete thing["b"]; benthos.v0_msg_set_structured(thing); })(); ``` ## [](#runtime)Runtime To optimize code execution, JavaScript runtimes are created on demand (in order to support parallel execution) and are reused across invocations. Therefore, it’s important to understand that the global state created by your programs will outlive individual invocations. For your programs to avoid failing after the first invocation, ensure that you do not define variables at the global scope. Although technically possible, it is recommended that you do not rely on the global state for maintaining state across invocations as the pooling nature of the runtimes will prevent deterministic behavior. We aim to support deterministic strategies for mutating global state in the future. ## [](#functions)Functions ### [](#benthos-v0_fetch)`benthos.v0_fetch` Executes an HTTP request synchronously and returns the result as an object of the form `{"status":200,"body":"foo"}`. #### [](#parameters)Parameters **`url`** The URL to fetch **`headers`** An object of string/string key/value pairs to add the request as headers. **`method`** The method of the request. **`body`** <(optional) string> A body to send. #### [](#examples-2)Examples ```javascript let result = benthos.v0_fetch("http://example.com", {}, "GET", "") benthos.v0_msg_set_structured(result); ``` ### [](#benthos-v0_msg_as_string)`benthos.v0_msg_as_string` Obtain the raw contents of the processed message as a string. #### [](#examples-3)Examples ```javascript let contents = benthos.v0_msg_as_string(); ``` ### [](#benthos-v0_msg_as_structured)`benthos.v0_msg_as_structured` Obtain the root of the processed message as a structured value. If the message is not valid JSON or has not already been expanded into a structured form this function will throw an error. #### [](#examples-4)Examples ```javascript let foo = benthos.v0_msg_as_structured().foo; ``` ### [](#benthos-v0_msg_exists_meta)`benthos.v0_msg_exists_meta` Check that a metadata key exists. #### [](#parameters-2)Parameters **`name`** The metadata key to search for. #### [](#examples-5)Examples ```javascript if (benthos.v0_msg_exists_meta("kafka_key")) {} ``` ### [](#benthos-v0_msg_get_meta)`benthos.v0_msg_get_meta` Get the value of a metadata key from the processed message. #### [](#parameters-3)Parameters **`name`** The metadata key to search for. #### [](#examples-6)Examples ```javascript let key = benthos.v0_msg_get_meta("kafka_key"); ``` ### [](#benthos-v0_msg_set_meta)`benthos.v0_msg_set_meta` Set a metadata key on the processed message to a value. #### [](#parameters-4)Parameters **`name`** The metadata key to set. **`value`** The value to set it to. #### [](#examples-7)Examples ```javascript benthos.v0_msg_set_meta("thing", "hello world"); ``` ### [](#benthos-v0_msg_set_string)`benthos.v0_msg_set_string` Set the contents of the processed message to a given string. #### [](#parameters-5)Parameters **`value`** The value to set it to. #### [](#examples-8)Examples ```javascript benthos.v0_msg_set_string("hello world"); ``` ### [](#benthos-v0_msg_set_structured)`benthos.v0_msg_set_structured` Set the root of the processed message to a given value of any type. #### [](#parameters-6)Parameters **`value`** The value to set it to. #### [](#examples-9)Examples ```javascript benthos.v0_msg_set_structured({ "foo": "a thing", "bar": "something else", "baz": 1234 }); ``` --- # Page 246: jira **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/jira.md --- # jira --- title: jira latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/jira page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/jira.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/jira.adoc description: Queries Jira resources and returns structured data. page-git-created-date: "2025-11-03" page-git-modified-date: "2026-01-05" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/jira/ "View the Cloud version of this component"), Self-Managed Queries Jira resources and returns structured data. Introduced in version 4.68.0. #### Common ```yaml processors: label: "" jira: username: "" # No default (required) api_token: "" # No default (required) max_results_per_page: 50 base_url: "" # No default (required) timeout: 5s ``` #### Advanced ```yaml processors: label: "" jira: username: "" # No default (required) api_token: "" # No default (required) max_results_per_page: 50 base_url: "" # No default (required) timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] proxy_url: "" disable_http2: false tps_limit: 0 tps_burst: 1 backoff: initial_interval: 1s max_interval: 30s max_retries: 3 tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s http: max_idle_conns: 100 max_idle_conns_per_host: 0 max_conns_per_host: 64 idle_conn_timeout: 1m30s tls_handshake_timeout: 10s expect_continue_timeout: 1s response_header_timeout: 0s disable_keep_alives: false disable_compression: false max_response_header_bytes: 1048576 max_response_body_bytes: 10485760 write_buffer_size: 4096 read_buffer_size: 4096 h2: strict_max_concurrent_requests: false max_decoder_header_table_size: 4096 max_encoder_header_table_size: 4096 max_read_frame_size: 16384 max_receive_buffer_per_connection: 1048576 max_receive_buffer_per_stream: 1048576 send_ping_timeout: 0s ping_timeout: 15s write_byte_timeout: 0s access_log_level: "" access_log_body_limit: 0 ``` Executes Jira API queries based on input messages and returns structured results. The processor handles pagination, retries, and field expansion automatically. Supports querying the following Jira resources: - Issues (JQL queries) - Issue transitions - Users - Roles - Project versions - Project categories - Project types - Projects The processor authenticates using basic authentication with username and API token. Input messages should contain valid Jira queries in JSON format. ## [](#fields)Fields ### [](#access_log_body_limit)`access_log_body_limit` Maximum bytes of request/response body to include in logs. 0 to skip body logging. **Type**: `int` **Default**: `0` ### [](#access_log_level)`access_log_level` Log level for HTTP request/response logging. Empty disables logging. **Type**: `string` **Default**: `""` **Options**: `` `, `TRACE ``, `DEBUG`, `INFO`, `WARN`, `ERROR` ### [](#api_token)`api_token` The Jira API token for the specified account. You can generate an API token from your [Atlassian account settings](https://id.atlassian.com/manage-profile/security/api-tokens). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#backoff)`backoff` Adaptive backoff configuration for 429 (Too Many Requests) responses. Always active. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` Initial interval between retries on 429 responses. **Type**: `string` **Default**: `1s` ### [](#backoff-max_interval)`backoff.max_interval` Maximum interval between retries on 429 responses. **Type**: `string` **Default**: `30s` ### [](#backoff-max_retries)`backoff.max_retries` Maximum number of retries on 429 responses. **Type**: `int` **Default**: `3` ### [](#base_url)`base_url` The base URL of the Jira instance (for example, `[https://your-domain.atlassian.net](https://your-domain.atlassian.net)`). **Type**: `string` ### [](#disable_http2)`disable_http2` Disable HTTP/2 and force HTTP/1.1. **Type**: `bool` **Default**: `false` ### [](#http)`http` HTTP transport settings controlling connection pooling, timeouts, and HTTP/2. **Type**: `object` ### [](#http-disable_compression)`http.disable_compression` Disable automatic decompression of gzip responses. **Type**: `bool` **Default**: `false` ### [](#http-disable_keep_alives)`http.disable_keep_alives` Disable HTTP keep-alive connections; each request uses a new connection. **Type**: `bool` **Default**: `false` ### [](#http-expect_continue_timeout)`http.expect_continue_timeout` Maximum time to wait for a server’s 100-continue response before sending the body. 0 means the body is sent immediately. **Type**: `string` **Default**: `1s` ### [](#http-h2)`http.h2` HTTP/2-specific transport settings. Only applied when HTTP/2 is enabled. **Type**: `object` ### [](#http-h2-max_decoder_header_table_size)`http.h2.max_decoder_header_table_size` Upper limit in bytes for the HPACK header table used to decode headers from the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-h2-max_encoder_header_table_size)`http.h2.max_encoder_header_table_size` Upper limit in bytes for the HPACK header table used to encode headers sent to the peer. Must be less than 4 MiB. **Type**: `int` **Default**: `4096` ### [](#http-h2-max_read_frame_size)`http.h2.max_read_frame_size` Largest HTTP/2 frame this endpoint will read. Valid range: 16 KiB to 16 MiB. **Type**: `int` **Default**: `16384` ### [](#http-h2-max_receive_buffer_per_connection)`http.h2.max_receive_buffer_per_connection` Maximum flow-control window size in bytes for data received on a connection. Must be at least 64 KiB and less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-h2-max_receive_buffer_per_stream)`http.h2.max_receive_buffer_per_stream` Maximum flow-control window size in bytes for data received on a single stream. Must be less than 4 MiB. **Type**: `int` **Default**: `1048576` ### [](#http-h2-ping_timeout)`http.h2.ping_timeout` Timeout waiting for a PING response before closing the connection. **Type**: `string` **Default**: `15s` ### [](#http-h2-send_ping_timeout)`http.h2.send_ping_timeout` Idle timeout after which a PING frame is sent to verify connection health. 0 disables health checks. **Type**: `string` **Default**: `0s` ### [](#http-h2-strict_max_concurrent_requests)`http.h2.strict_max_concurrent_requests` When true, new requests block when a connection’s concurrency limit is reached instead of opening a new connection. **Type**: `bool` **Default**: `false` ### [](#http-h2-write_byte_timeout)`http.h2.write_byte_timeout` Timeout for writing data to a connection. The timer resets whenever bytes are written. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-idle_conn_timeout)`http.idle_conn_timeout` How long an idle connection remains in the pool before being closed. 0 disables the timeout. **Type**: `string` **Default**: `1m30s` ### [](#http-max_conns_per_host)`http.max_conns_per_host` Maximum total connections (active + idle) per host. 0 means unlimited. **Type**: `int` **Default**: `64` ### [](#http-max_idle_conns)`http.max_idle_conns` Maximum total number of idle (keep-alive) connections across all hosts. 0 means unlimited. **Type**: `int` **Default**: `100` ### [](#http-max_idle_conns_per_host)`http.max_idle_conns_per_host` Maximum idle connections to keep per host. 0 (the default) uses GOMAXPROCS+1. **Type**: `int` **Default**: `0` ### [](#http-max_response_body_bytes)`http.max_response_body_bytes` Maximum bytes of response body the client will read. The response body is wrapped with a limit reader; reads beyond this cap return EOF. 0 disables the limit. **Type**: `int` **Default**: `10485760` ### [](#http-max_response_header_bytes)`http.max_response_header_bytes` Maximum bytes of response headers to allow. **Type**: `int` **Default**: `1048576` ### [](#http-read_buffer_size)`http.read_buffer_size` Size in bytes of the per-connection read buffer. **Type**: `int` **Default**: `4096` ### [](#http-response_header_timeout)`http.response_header_timeout` Maximum time to wait for response headers after writing the full request. 0 disables the timeout. **Type**: `string` **Default**: `0s` ### [](#http-tls_handshake_timeout)`http.tls_handshake_timeout` Maximum time to wait for a TLS handshake to complete. 0 disables the timeout. **Type**: `string` **Default**: `10s` ### [](#http-write_buffer_size)`http.write_buffer_size` Size in bytes of the per-connection write buffer. **Type**: `int` **Default**: `4096` ### [](#max_results_per_page)`max_results_per_page` The maximum number of results to return per page when calling the Jira API. [Pagination](https://docs.atlassian.com/software/jira/docs/api/REST/9.17.0/#pagination) in the Jira API is zero-based, so the first page starts at `0`. **Type**: `int` **Default**: `50` ### [](#proxy_url)`proxy_url` HTTP proxy URL. Empty string disables proxying. **Type**: `string` **Default**: `""` ### [](#tcp)`tcp` TCP socket configuration. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` HTTP request timeout. **Type**: `string` **Default**: `5s` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#tps_burst)`tps_burst` Maximum burst size for rate limiting. **Type**: `int` **Default**: `1` ### [](#tps_limit)`tps_limit` Rate limit in requests per second. 0 disables rate limiting. **Type**: `float` **Default**: `0` ### [](#username)`username` The username or email address of the Jira account. **Type**: `string` ## [](#examples)Examples ### [](#minimal-configuration)Minimal configuration Basic Jira processor setup with required fields only ```yaml pipeline: processors: - jira: base_url: "https://your-domain.atlassian.net" username: "${JIRA_USERNAME}" api_token: "${JIRA_API_TOKEN}" ``` ### [](#full-configuration-with-tuning)Full configuration with tuning Complete configuration with pagination and timeout settings ```yaml pipeline: processors: - jira: base_url: "https://your-domain.atlassian.net" username: "${JIRA_USERNAME}" api_token: "${JIRA_API_TOKEN}" max_results_per_page: 200 timeout: "30s" ``` ## Suggested labs - [Stream Jira Issues to Redpanda for Real-Time Metrics](/redpanda-labs/docker-compose/jira-metrics-pipeline/) [Search all labs](/redpanda-labs) --- # Page 247: jmespath **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/jmespath.md --- # jmespath --- title: jmespath latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/jmespath page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/jmespath.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/jmespath.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/jmespath/ "View the Cloud version of this component"), Self-Managed Executes a [JMESPath query](http://jmespath.org/) on JSON documents and replaces the message with the resulting document. ```yml # Config fields, showing default values label: "" jmespath: query: "" # No default (required) ``` > 💡 **TIP: Try out Bloblang** > > Try out Bloblang > > For better performance and improved capabilities try native Redpanda Connect mapping with the [`mapping` processor](../mapping/). ## [](#fields)Fields ### [](#query)`query` The JMESPath query to apply to messages. **Type**: `string` nclude::redpanda-connect:components:partial$examples/processors/jmespath.adoc\[\] --- # Page 248: jq **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/jq.md --- # jq --- title: jq latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/jq page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/jq.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/jq.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/jq/ "View the Cloud version of this component"), Self-Managed Transforms and filters messages using jq queries. #### Common ```yml processors: label: "" jq: query: "" # No default (required) ``` #### Advanced ```yml processors: label: "" jq: query: "" # No default (required) raw: false output_raw: false ``` > 💡 **TIP: Try out Bloblang** > > Try out Bloblang > > For better performance and improved capabilities try out native Redpanda Connect mapping with the [`mapping` processor](../mapping/). The provided query is executed on each message, targeting either the contents as a structured JSON value or as a raw string using the field `raw`, and the message is replaced with the query result. Message metadata is also accessible within the query from the variable `$metadata`. This processor uses the [gojq library](https://github.com/itchyny/gojq), and therefore does not require jq to be installed as a dependency. However, this also means there are some [differences in how these queries are executed](https://github.com/itchyny/gojq#difference-to-jq) versus the jq cli. If the query does not emit any value then the message is filtered, if the query returns multiple values then the resulting message will be an array containing all values. The full query syntax is described in [jq’s documentation](https://stedolan.github.io/jq/manual/). ## [](#error-handling)Error handling Queries can fail, in which case the message remains unchanged, errors are logged, and the message is flagged as having failed, allowing you to use [standard processor error handling patterns](../../../configuration/error_handling/). ## [](#fields)Fields ### [](#output_raw)`output_raw` Whether to output raw text (unquoted) instead of JSON strings when the emitted values are string types. **Type**: `bool` **Default**: `false` ### [](#query)`query` The jq query to filter and transform messages with. **Type**: `string` ### [](#raw)`raw` Whether to process the input as a raw string instead of as JSON. **Type**: `bool` **Default**: `false` ## [](#examples)Examples ### [](#mapping)Mapping When receiving JSON documents of the form: ```json { "locations": [ {"name": "Seattle", "state": "WA"}, {"name": "New York", "state": "NY"}, {"name": "Bellevue", "state": "WA"}, {"name": "Olympia", "state": "WA"} ] } ``` We could collapse the location names from the state of Washington into a field `Cities`: ```json {"Cities": "Bellevue, Olympia, Seattle"} ``` With the following config: ```yaml pipeline: processors: - jq: query: '{Cities: .locations | map(select(.state == "WA").name) | sort | join(", ") }' ``` --- # Page 249: json_schema **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/json_schema.md --- # json\_schema --- title: json_schema latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/json_schema page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/json_schema.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/json_schema.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/json_schema/ "View the Cloud version of this component"), Self-Managed Checks messages against a provided JSONSchema definition but does not change the payload under any circumstances. If a message does not match the schema it can be caught using [error handling methods](../../../configuration/error_handling/). ```yml # Config fields, showing default values label: "" json_schema: schema: "" # No default (optional) schema_path: "" # No default (optional) ``` Please refer to the [JSON Schema website](https://json-schema.org/) for information and tutorials regarding the syntax of the schema. ## [](#fields)Fields ### [](#schema)`schema` A schema to apply. Use either this or the `schema_path` field. **Type**: `string` ### [](#schema_path)`schema_path` The path of a schema document to apply. Use either this or the `schema` field. **Type**: `string` ## [](#examples)Examples With the following JSONSchema document: ```json { "$id": "https://example.com/person.schema.json", "$schema": "http://json-schema.org/draft-07/schema#", "title": "Person", "type": "object", "properties": { "firstName": { "type": "string", "description": "The person's first name." }, "lastName": { "type": "string", "description": "The person's last name." }, "age": { "description": "Age in years which must be equal to or greater than zero.", "type": "integer", "minimum": 0 } } } ``` And the following Redpanda Connect configuration: ```yaml pipeline: processors: - json_schema: schema_path: "file://path_to_schema.json" - catch: - log: level: ERROR message: "Schema validation failed due to: ${!error()}" - mapping: 'root = deleted()' # Drop messages that fail ``` If a payload being processed looked like: ```json {"firstName":"John","lastName":"Doe","age":-21} ``` Then a log message would appear explaining the fault and the payload would be dropped. --- # Page 250: log **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/log.md --- # log --- title: log latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/log page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/log.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/log.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/log/ "View the Cloud version of this component"), Self-Managed Prints a log event for each message. Messages always remain unchanged. The log message can be set using function interpolations described in [Bloblang queries](../../../configuration/interpolation/#bloblang-queries) which allows you to log the contents and metadata of messages. ```yml # Config fields, showing default values label: "" log: level: INFO fields_mapping: |- # No default (optional) root.reason = "cus I wana" root.id = this.id root.age = this.user.age.number() root.kafka_topic = meta("kafka_topic") message: "" ``` The `level` field determines the log level of the printed events and can be any of the following values: TRACE, DEBUG, INFO, WARN, ERROR. ## [](#structured-fields)Structured fields It’s also possible add custom fields to logs when the format is set to a structured form such as `json` or `logfmt` with the config field [`fields_mapping`](#fields_mapping): ```yaml pipeline: processors: - log: level: DEBUG message: hello world fields_mapping: | root.reason = "cus I wana" root.id = this.id root.age = this.user.age root.kafka_topic = meta("kafka_topic") ``` ## [](#fields)Fields ### [](#fields_mapping)`fields_mapping` An optional [Bloblang mapping](../../../guides/bloblang/about/) that can be used to specify extra fields to add to the log. If log fields are also added with the `fields` field then those values will override matching keys from this mapping. **Type**: `string` ```yaml # Examples: fields_mapping: |- root.reason = "cus I wana" root.id = this.id root.age = this.user.age.number() root.kafka_topic = meta("kafka_topic") ``` ### [](#level)`level` The log level to use. **Type**: `string` **Default**: `INFO` **Options**: `ERROR`, `WARN`, `INFO`, `DEBUG`, `TRACE` ### [](#message)`message` The message to print. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` --- # Page 251: mapping **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/mapping.md --- # mapping --- title: mapping latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/mapping page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/mapping.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/mapping.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-02-04" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/mapping/ "View the Cloud version of this component"), Self-Managed Executes a [Bloblang](../../../guides/bloblang/about/) mapping on messages, creating a new document that replaces (or filters) the original message. Introduced in version 4.5.0. ```yml # Config fields, showing default values label: "" mapping: "" # No default (required) ``` Bloblang is a powerful language that enables a wide range of mapping, transformation and filtering tasks. For more information, see [Bloblang](../../../guides/bloblang/about/). If your mapping is large and you’d prefer for it to live in a separate file then you can execute a mapping directly from a file with the expression `from ""`, where the path must be absolute, or relative from the location that Redpanda Connect is executed from. Note: This processor is equivalent to the [Bloblang](../bloblang/#component-rename) one. The latter will be deprecated in a future release. ## [](#input-document-immutability)Input document immutability Mapping operates by creating an entirely new object during assignments, this has the advantage of treating the original referenced document as immutable and therefore queryable at any stage of your mapping. For example, with the following mapping: ```bloblang root.id = this.id root.invitees = this.invitees.filter(i -> i.mood >= 0.5) root.rejected = this.invitees.filter(i -> i.mood < 0.5) # In: {"id":"party-2024","invitees":[{"name":"Alice","mood":0.8},{"name":"Bob","mood":0.3},{"name":"Carol","mood":0.9}]} ``` Notice that we mutate the value of `invitees` in the resulting document by filtering out objects with a lower mood. However, even after doing so we’re still able to reference the unchanged original contents of this value from the input document in order to populate a second field. Within this mapping we also have the flexibility to reference the mutable mapped document by using the keyword `root` (i.e. `root.invitees`) on the right-hand side instead. Mapping documents is advantageous in situations where the result is a document with a dramatically different shape to the input document, since we are effectively rebuilding the document in its entirety and might as well keep a reference to the unchanged input document throughout. However, in situations where we are only performing minor alterations to the input document, the rest of which is unchanged, it might be more efficient to use the [`mutation` processor](../mutation/) instead. ## [](#error-handling)Error handling Bloblang mappings can fail, in which case the message remains unchanged, errors are logged, and the message is flagged as having failed, allowing you to use [standard processor error handling patterns](../../../configuration/error_handling/). However, Bloblang itself also provides powerful ways of ensuring your mappings do not fail by specifying desired [fallback behavior](../../../guides/bloblang/about/#error-handling). ## [](#examples)Examples ### [](#mapping)Mapping Given JSON documents containing an array of fans: ```json { "id":"foo", "description":"a show about foo", "fans":[ {"name":"bev","obsession":0.57}, {"name":"grace","obsession":0.21}, {"name":"ali","obsession":0.89}, {"name":"vic","obsession":0.43} ] } ``` We can reduce the documents down to just the ID and only those fans with an obsession score above 0.5, giving us: ```json { "id":"foo", "fans":[ {"name":"bev","obsession":0.57}, {"name":"ali","obsession":0.89} ] } ``` With the following config: ```yaml pipeline: processors: - mapping: | root.id = this.id root.fans = this.fans.filter(fan -> fan.obsession > 0.5) ``` ### [](#more-mapping)More Mapping When receiving JSON documents of the form: ```json { "locations": [ {"name": "Seattle", "state": "WA"}, {"name": "New York", "state": "NY"}, {"name": "Bellevue", "state": "WA"}, {"name": "Olympia", "state": "WA"} ] } ``` We could collapse the location names from the state of Washington into a field `Cities`: ```json {"Cities": "Bellevue, Olympia, Seattle"} ``` With the following config: ```yaml pipeline: processors: - mapping: | root.Cities = this.locations. filter(loc -> loc.state == "WA"). map_each(loc -> loc.name). sort().join(", ") ``` --- # Page 252: metric **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/metric.md --- # metric --- title: metric latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/metric page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/metric.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/metric.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/metric/ "View the Cloud version of this component"), Self-Managed Emit custom metrics by extracting values from messages. ```yml # Config fields, showing default values label: "" metric: type: "" # No default (required) name: "" # No default (required) labels: {} # No default (optional) value: "" ``` This processor works by evaluating an [interpolated field `value`](../../../configuration/interpolation/#bloblang-queries) for each message and updating a emitted metric according to the [type](#types). Custom metrics such as these are emitted along with Redpanda Connect internal metrics, where you can customize where metrics are sent, which metric names are emitted and rename them as/when appropriate. For more information see the [metrics docs](../../metrics/about/). ## [](#fields)Fields ### [](#labels)`labels` A map of label names and values that can be used to enrich metrics. Labels are not supported by some metric destinations, in which case the metrics series are combined. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: labels: topic: ${! meta("kafka_topic") } type: ${! json("doc.type") } ``` ### [](#name)`name` The name of the metric to create, this must be unique across all Redpanda Connect components otherwise it will overwrite those other metrics. **Type**: `string` ### [](#type)`type` The metric [type](#types) to create. **Type**: `string` **Options**: `counter`, `counter_by`, `gauge`, `timing` ### [](#value)`value` For some metric types specifies a value to set, increment. Certain metrics exporters such as Prometheus support floating point values, but those that do not will cast a floating point value into an integer. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `""` ## [](#examples)Examples ### [](#counter)Counter In this example we emit a counter metric called `Foos`, which increments for every message processed, and we label the metric with some metadata about where the message came from and a field from the document that states what type it is. We also configure our metrics to emit to CloudWatch, and explicitly only allow our custom metric and some internal Redpanda Connect metrics to emit. ```yaml pipeline: processors: - metric: name: Foos type: counter labels: topic: ${! meta("kafka_topic") } partition: ${! meta("kafka_partition") } type: ${! json("document.type").or("unknown") } metrics: mapping: | root = if ![ "Foos", "input_received", "output_sent" ].contains(this) { deleted() } aws_cloudwatch: namespace: ProdConsumer ``` ### [](#gauge)Gauge In this example we emit a gauge metric called `FooSize`, which is given a value extracted from JSON messages at the path `foo.size`. We then also configure our Prometheus metric exporter to only emit this custom metric and nothing else. We also label the metric with some metadata. ```yaml pipeline: processors: - metric: name: FooSize type: gauge labels: topic: ${! meta("kafka_topic") } value: ${! json("foo.size") } metrics: mapping: 'if this != "FooSize" { deleted() }' prometheus: {} ``` ## [](#types)Types ### [](#counter-2)`counter` Increments a counter by exactly 1, the contents of `value` are ignored by this type. ### [](#counter_by)`counter_by` If the contents of `value` can be parsed as a positive integer value then the counter is incremented by this value. For example, the following configuration will increment the value of the `count.custom.field` metric by the contents of `field.some.value`: ```yaml pipeline: processors: - metric: type: counter_by name: CountCustomField value: ${!json("field.some.value")} ``` ### [](#gauge-2)`gauge` If the contents of `value` can be parsed as a positive integer value then the gauge is set to this value. For example, the following configuration will set the value of the `gauge.custom.field` metric to the contents of `field.some.value`: ```yaml pipeline: processors: - metric: type: gauge name: GaugeCustomField value: ${!json("field.some.value")} ``` ### [](#timing)`timing` Equivalent to `gauge` where instead the metric is a timing. It is recommended that timing values are recorded in nanoseconds in order to be consistent with standard Redpanda Connect timing metrics, as in some cases these values are automatically converted into other units such as when exporting timings as histograms with Prometheus metrics. --- # Page 253: mongodb **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/mongodb.md --- # mongodb --- title: mongodb latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/mongodb page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/mongodb.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/mongodb.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Processor ▼ [Processor](/redpanda-connect/components/processors/mongodb/)[Cache](/redpanda-connect/components/caches/mongodb/)[Input](/redpanda-connect/components/inputs/mongodb/)[Output](/redpanda-connect/components/outputs/mongodb/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/mongodb/ "View the Cloud version of this component"), Self-Managed Performs operations against MongoDB for each message, allowing you to store or retrieve data within message payloads. Introduced in version 3.43.0. #### Common ```yml processors: label: "" mongodb: url: "" # No default (required) database: "" # No default (required) username: "" password: "" collection: "" # No default (required) operation: insert-one write_concern: w: majority j: false w_timeout: "" document_map: "" filter_map: "" hint_map: "" upsert: false ``` #### Advanced ```yml processors: label: "" mongodb: url: "" # No default (required) database: "" # No default (required) username: "" password: "" app_name: benthos collection: "" # No default (required) operation: insert-one write_concern: w: majority j: false w_timeout: "" document_map: "" filter_map: "" hint_map: "" upsert: false json_marshal_mode: canonical ``` ## [](#fields)Fields ### [](#app_name)`app_name` The client application name. **Type**: `string` **Default**: `benthos` ### [](#collection)`collection` The name of the target collection. **Type**: `string` ### [](#database)`database` The name of the target MongoDB database. **Type**: `string` ### [](#document_map)`document_map` A Bloblang map that represents a document to store in MongoDB, expressed as [extended JSON in canonical form](https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/). The `document_map` parameter is required for the following database operations: `insert-one`, `replace-one`, `update-one`, and `aggregate`. **Type**: `string` **Default**: `""` ```yaml # Examples: document_map: |- root.a = this.foo root.b = this.bar ``` ### [](#filter_map)`filter_map` A Bloblang map that represents a filter for a MongoDB command, expressed as [extended JSON in canonical form](https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/). The `filter_map` parameter is required for all database operations except `insert-one`. This output uses `filter_map` to find documents for the specified operation. For example, for a `delete-one` operation, the filter map should include the fields required to locate the document for deletion. **Type**: `string` **Default**: `""` ```yaml # Examples: filter_map: |- root.a = this.foo root.b = this.bar ``` ### [](#hint_map)`hint_map` A Bloblang map that represents a hint or index for a MongoDB command to use, expressed as [extended JSON in canonical form](https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/). This map is optional, and is used with all operations except `insert-one`. Define a `hint_map` to improve performance when finding documents in the MongoDB database. **Type**: `string` **Default**: `""` ```yaml # Examples: hint_map: |- root.a = this.foo root.b = this.bar ``` ### [](#json_marshal_mode)`json_marshal_mode` Controls the format of the output message (optional). Requires version 3.60.0 or later. **Type**: `string` **Default**: `canonical` | Option | Summary | | --- | --- | | canonical | A string format that emphasizes type preservation at the expense of readability and interoperability. That is, conversion from canonical to BSON will generally preserve type information except in certain specific cases. | | relaxed | A string format that emphasizes readability and interoperability at the expense of type preservation. That is, conversion from relaxed format to BSON can lose type information. | ### [](#operation)`operation` The MongoDB database operation to perform. **Type**: `string` **Default**: `insert-one` **Options**: `insert-one`, `delete-one`, `delete-many`, `replace-one`, `update-one`, `find-one`, `aggregate` ### [](#password)`password` The password to use for authentication. Used together with `username` for basic authentication or with encrypted private keys for secure access. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#upsert)`upsert` The `upsert` parameter is optional, and only applies for `update-one` and `replace-one` operations. If the filter specified in `filter_map` matches an existing document, this operation updates or replaces the document, otherwise a new document is created. Requires version 3.60.0 or later. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target MongoDB server. **Type**: `string` ```yaml # Examples: url: mongodb://localhost:27017 ``` ### [](#username)`username` The username required to connect to the database. **Type**: `string` **Default**: `""` ### [](#write_concern)`write_concern` The [write concern settings](https://www.mongodb.com/docs/manual/reference/write-concern/) for the MongoDB connection. **Type**: `object` ### [](#write_concern-j)`write_concern.j` The `j` requests acknowledgement from MongoDB, which is created when write operations are written to the journal. **Type**: `bool` **Default**: `false` ### [](#write_concern-w)`write_concern.w` The `w` requests acknowledgement, which write operations propagate to the specified number of MongoDB instances. **Type**: `string` **Default**: `majority` ### [](#write_concern-w_timeout)`write_concern.w_timeout` The write concern timeout. **Type**: `string` **Default**: `""` --- # Page 254: msgpack **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/msgpack.md --- # msgpack --- title: msgpack latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/msgpack page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/msgpack.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/msgpack.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Available in:** Self-Managed Converts messages to or from the [MessagePack](https://msgpack.org/) format. Introduced in version 3.59.0. ```yml # Config fields, showing default values label: "" msgpack: operator: "" # No default (required) ``` ## [](#fields)Fields ### [](#operator)`operator` The operation to perform on messages. **Type**: `string` | Option | Summary | | --- | --- | | from_json | Convert JSON messages to MessagePack format | | to_json | Convert MessagePack messages to JSON format | --- # Page 255: mutation **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/mutation.md --- # mutation --- title: mutation latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/mutation page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/mutation.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/mutation.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-02-04" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/mutation/ "View the Cloud version of this component"), Self-Managed Executes a [Bloblang](../../../guides/bloblang/about/) mapping and directly transforms the contents of messages, mutating (or deleting) them. Introduced in version 4.5.0. ```yml # Config fields, showing default values label: "" mutation: "" # No default (required) ``` Bloblang is a powerful language that enables a wide range of mapping, transformation and filtering tasks. For more information, see [Bloblang](../../../guides/bloblang/about/). If your mapping is large and you’d prefer for it to live in a separate file then you can execute a mapping directly from a file with the expression `from ""`, where the path must be absolute, or relative from the location that Redpanda Connect is executed from. ## [](#input-document-mutability)Input document mutability A mutation is a mapping that transforms input documents directly, this has the advantage of reducing the need to copy the data fed into the mapping. However, this also means that the referenced document is mutable and therefore changes throughout the mapping. For example, with the following Bloblang: ```bloblang root.rejected = this.invitees.filter(i -> i.mood < 0.5) root.invitees = this.invitees.filter(i -> i.mood >= 0.5) # In: {"invitees":[{"name":"Alice","mood":0.8},{"name":"Bob","mood":0.3},{"name":"Carol","mood":0.9}]} ``` Notice that we create a field `rejected` by copying the array field `invitees` and filtering out objects with a high mood. We then overwrite the field `invitees` by filtering out objects with a low mood, resulting in two array fields that are each a subset of the original. If we were to reverse the ordering of these assignments like so: ```bloblang root.invitees = this.invitees.filter(i -> i.mood >= 0.5) root.rejected = this.invitees.filter(i -> i.mood < 0.5) # In: {"invitees":[{"name":"Alice","mood":0.8},{"name":"Bob","mood":0.3},{"name":"Carol","mood":0.9}]} ``` Then the new field `rejected` would be empty as we have already mutated `invitees` to exclude the objects that it would be populated by. We can solve this problem either by carefully ordering our assignments or by capturing the original array using a variable (`let invitees = this.invitees`). Mutations are advantageous over a standard mapping in situations where the result is a document with mostly the same shape as the input document, since we can avoid unnecessarily copying data from the referenced input document. However, in situations where we are creating an entirely new document shape it can be more convenient to use the traditional [`mapping` processor](../mapping/) instead. ## [](#error-handling)Error handling Bloblang mappings can fail, in which case the error is logged and the message is flagged as having failed, allowing you to use [standard processor error handling patterns](../../../configuration/error_handling/). However, Bloblang itself also provides powerful ways of ensuring your mappings do not fail by specifying desired [fallback behavior](../../../guides/bloblang/about/#error-handling). ## [](#examples)Examples ### [](#mapping)Mapping Given JSON documents containing an array of fans: ```json { "id":"foo", "description":"a show about foo", "fans":[ {"name":"bev","obsession":0.57}, {"name":"grace","obsession":0.21}, {"name":"ali","obsession":0.89}, {"name":"vic","obsession":0.43} ] } ``` We can reduce the documents down to just the ID and only those fans with an obsession score above 0.5, giving us: ```json { "id":"foo", "fans":[ {"name":"bev","obsession":0.57}, {"name":"ali","obsession":0.89} ] } ``` With the following config: ```yaml pipeline: processors: - mutation: | root.description = deleted() root.fans = this.fans.filter(fan -> fan.obsession > 0.5) ``` ### [](#more-mapping)More Mapping When receiving JSON documents of the form: ```json { "locations": [ {"name": "Seattle", "state": "WA"}, {"name": "New York", "state": "NY"}, {"name": "Bellevue", "state": "WA"}, {"name": "Olympia", "state": "WA"} ] } ``` We could collapse the location names from the state of Washington into a field `Cities`: ```json {"Cities": "Bellevue, Olympia, Seattle"} ``` With the following config: ```yaml pipeline: processors: - mutation: | root.Cities = this.locations. filter(loc -> loc.state == "WA"). map_each(loc -> loc.name). sort().join(", ") ``` --- # Page 256: nats_kv **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/nats_kv.md --- # nats\_kv --- title: nats_kv latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/nats_kv page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/nats_kv.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/nats_kv.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Processor ▼ [Processor](/redpanda-connect/components/processors/nats_kv/)[Cache](/redpanda-connect/components/caches/nats_kv/)[Input](/redpanda-connect/components/inputs/nats_kv/)[Output](/redpanda-connect/components/outputs/nats_kv/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/nats_kv/ "View the Cloud version of this component"), Self-Managed Perform operations on a NATS key-value bucket. Introduced in version 4.12.0. #### Common ```yml processors: label: "" nats_kv: urls: [] # No default (required) bucket: "" # No default (required) operation: "" # No default (required) key: "" # No default (required) ``` #### Advanced ```yml processors: label: "" nats_kv: urls: [] # No default (required) max_reconnects: "" # No default (optional) bucket: "" # No default (required) operation: "" # No default (required) key: "" # No default (required) revision: "" # No default (optional) timeout: 5s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) ``` ## [](#kv-operations)KV operations The NATS KV processor supports many KV operations using the [`operation`](#operation) field. Along with `get`, `put`, and `delete`, this processor supports atomic operations like `update` and `create`, as well as utility operations like `purge`, `history`, and `keys`. ## [](#metadata)Metadata This processor adds the following metadata fields to each message, depending on the chosen `operation`: ### [](#get-get_revision)get, get\_revision ```text - nats_kv_key - nats_kv_bucket - nats_kv_revision - nats_kv_delta - nats_kv_operation - nats_kv_created ``` ### [](#create-update-delete-purge)create, update, delete, purge ```text - nats_kv_key - nats_kv_bucket - nats_kv_revision - nats_kv_operation ``` ### [](#keys)keys ```text - nats_kv_bucket ``` ## [](#connection-name)Connection name When monitoring and managing a production [NATS system](https://docs.nats.io/nats-concepts/overview), it is often useful to know which connection a message was sent or received from. To achieve this, set the connection name option when creating a NATS connection. Redpanda Connect can then automatically set the connection name to the NATS component label, so that monitoring tools between NATS and Redpanda Connect can stay in sync. ## [](#authentication)Authentication A number of Redpanda Connect components use NATS services. Each of these components support optional, advanced authentication parameters for [NKeys](https://docs.nats.io/nats-server/configuration/securing_nats/auth_intro/nkey_auth) and [user credentials](https://docs.nats.io/using-nats/developer/connecting/creds). For an in-depth guide, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/nats_admin/security/jwt). ### [](#nkeys)NKeys NATS server can use NKeys in several ways for authentication. The simplest approach is to configure the server with a list of user’s public keys. The server can then generate a challenge for each connection request from a client, and the client must respond to the challenge by signing it with its private NKey, configured in the `nkey_file` or `nkey` field. For more details, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth). ### [](#user-credentials)User credentials NATS server also supports decentralized authentication based on JSON Web Tokens (JWTs). When a server is configured to use this authentication scheme, clients need a [user JWT](https://docs.nats.io/nats-server/configuration/securing_nats/jwt#json-web-tokens) and a corresponding [NKey secret](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth) to connect. You can use either of the following methods to supply the user JWT and NKey secret: - In the `user_credentials_file` field, enter the path to a file containing both the private key and the JWT. You can generate the file using the [nsc tool](https://docs.nats.io/nats-tools/nsc). - In the `user_jwt` field, enter a plain text JWT, and in the `user_nkey_seed` field, enter the plain text NKey seed or private key. For more details about authentication using JWTs, see the [NATS documentation](https://docs.nats.io/using-nats/developer/connecting/creds). ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed or private key for NATS authentication. NKeys provide secure, cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the corresponding user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#bucket)`bucket` The name of the KV bucket. **Type**: `string` ```yaml # Examples: bucket: my_kv_bucket ``` ### [](#key)`key` The key for each message. Supports [wildcards](https://docs.nats.io/nats-concepts/subjects#wildcards) for the `history` and `keys` operations. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: key: foo # --- key: foo.bar.baz # --- key: foo.* # --- key: foo.> # --- key: foo.${! json("meta.type") } ``` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. **Type**: `int` ### [](#operation)`operation` The operation to perform on the KV bucket. **Type**: `string` | Option | Summary | | --- | --- | | create | Adds the key/value pair if it does not exist. Returns an error if it already exists. | | delete | Deletes the key/value pair, but keeps historical values. | | get | Returns the latest value for key. | | get_revision | Returns the value of key for the specified revision. | | history | Returns historical values of key as an array of objects containing the following fields: key, value, bucket, revision, delta, operation, created. | | keys | Returns the keys in the bucket which match the keys_filter as an array of strings. | | purge | Deletes the key/value pair and all historical values. | | put | Places a new value for the key into the store. | | update | Updates the value for key only if the revision matches the latest revision. | ### [](#revision)`revision` The revision of the key to operate on. Used for `get_revision` and `update` operations. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: revision: 42 # --- revision: ${! @nats_kv_revision } ``` ### [](#timeout)`timeout` The maximum period to wait on an operation before aborting and returning an error. **Type**: `string` **Default**: `5s` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. Requires version 4.60.0 or later. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 257: nats_request_reply **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/nats_request_reply.md --- # nats\_request\_reply --- title: nats_request_reply latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/nats_request_reply page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/nats_request_reply.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/nats_request_reply.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/nats_request_reply/ "View the Cloud version of this component"), Self-Managed Sends a message to a NATS subject and expects a reply back from a NATS subscriber acting as a responder. Introduced in version 4.27.0. #### Common ```yml processors: label: "" nats_request_reply: urls: [] # No default (required) subject: "" # No default (required) headers: {} metadata: include_prefixes: [] include_patterns: [] timeout: 3s ``` #### Advanced ```yml processors: label: "" nats_request_reply: urls: [] # No default (required) max_reconnects: "" # No default (optional) subject: "" # No default (required) inbox_prefix: "" # No default (optional) headers: {} metadata: include_prefixes: [] include_patterns: [] timeout: 3s tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] tls_handshake_first: false auth: nkey_file: "" # No default (optional) nkey: "" # No default (optional) user_credentials_file: "" # No default (optional) user_jwt: "" # No default (optional) user_nkey_seed: "" # No default (optional) user: "" # No default (optional) password: "" # No default (optional) token: "" # No default (optional) ``` ## [](#metadata)Metadata This input adds the following metadata fields to each message: ```text - nats_subject - nats_sequence_stream - nats_sequence_consumer - nats_num_delivered - nats_num_pending - nats_domain - nats_timestamp_unix_nano ``` You can access these metadata fields using [function interpolation](../../../configuration/interpolation/#bloblang-queries). ## [](#connection-name)Connection name When monitoring and managing a production [NATS system](https://docs.nats.io/nats-concepts/overview), it is often useful to know which connection a message was sent or received from. To achieve this, set the connection name option when creating a NATS connection. Redpanda Connect can then automatically set the connection name to the NATS component label, so that monitoring tools between NATS and Redpanda Connect can stay in sync. ## [](#authentication)Authentication A number of Redpanda Connect components use NATS services. Each of these components support optional, advanced authentication parameters for [NKeys](https://docs.nats.io/nats-server/configuration/securing_nats/auth_intro/nkey_auth) and [user credentials](https://docs.nats.io/using-nats/developer/connecting/creds). For an in-depth guide, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/nats_admin/security/jwt). ### [](#nkeys)NKeys NATS server can use NKeys in several ways for authentication. The simplest approach is to configure the server with a list of user’s public keys. The server can then generate a challenge for each connection request from a client, and the client must respond to the challenge by signing it with its private NKey, configured in the `nkey_file` or `nkey` field. For more details, see the [NATS documentation](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth). ### [](#user-credentials)User credentials NATS server also supports decentralized authentication based on JSON Web Tokens (JWTs). When a server is configured to use this authentication scheme, clients need a [user JWT](https://docs.nats.io/nats-server/configuration/securing_nats/jwt#json-web-tokens) and a corresponding [NKey secret](https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/nkey_auth) to connect. You can use either of the following methods to supply the user JWT and NKey secret: - In the `user_credentials_file` field, enter the path to a file containing both the private key and the JWT. You can generate the file using the [nsc tool](https://docs.nats.io/nats-tools/nsc). - In the `user_jwt` field, enter a plain text JWT, and in the `user_nkey_seed` field, enter the plain text NKey seed or private key. For more details about authentication using JWTs, see the [NATS documentation](https://docs.nats.io/using-nats/developer/connecting/creds). ## [](#fields)Fields ### [](#auth)`auth` Optional configuration of NATS authentication parameters. **Type**: `object` ### [](#auth-nkey)`auth.nkey` Your NKey seed or private key for NATS authentication. NKeys provide secure, cryptographic authentication without passwords. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 ``` ### [](#auth-nkey_file)`auth.nkey_file` An optional file containing a NKey seed. **Type**: `string` ```yaml # Examples: nkey_file: ./seed.nk ``` ### [](#auth-password)`auth.password` An optional plain text password (given along with the corresponding user name). > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-token)`auth.token` An optional plain text token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-user)`auth.user` An optional plain text user name (given along with the corresponding user password). **Type**: `string` ### [](#auth-user_credentials_file)`auth.user_credentials_file` An optional file containing user credentials which consist of a user JWT and corresponding NKey seed. **Type**: `string` ```yaml # Examples: user_credentials_file: ./user.creds ``` ### [](#auth-user_jwt)`auth.user_jwt` An optional plaintext user JWT to use along with the corresponding user NKey seed. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#auth-user_nkey_seed)`auth.user_nkey_seed` An optional plaintext user NKey seed to use along with the corresponding user JWT. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#headers)`headers` Explicit message headers to add to messages. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` **Default**: `{}` ```yaml # Examples: headers: Content-Type: application/json Timestamp: ${!meta("Timestamp")} ``` ### [](#inbox_prefix)`inbox_prefix` Set an explicit inbox prefix for the response subject **Type**: `string` ```yaml # Examples: inbox_prefix: _INBOX_joe ``` ### [](#max_reconnects)`max_reconnects` The maximum number of times to attempt to reconnect to the server. If negative, it will never stop trying to reconnect. **Type**: `int` ### [](#metadata-2)`metadata` Determine which (if any) metadata values should be added to messages as headers. **Type**: `object` ### [](#metadata-include_patterns)`metadata.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#metadata-include_prefixes)`metadata.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#subject)`subject` A subject to write to. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: subject: foo.bar.baz # --- subject: ${! meta("kafka_topic") } # --- subject: foo.${! json("meta.type") } ``` ### [](#timeout)`timeout` A duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, such as 300ms, -1.5h or 2h45m. Valid time units are ns, us (or µs), ms, s, m, h. **Type**: `string` **Default**: `3s` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#tls_handshake_first)`tls_handshake_first` Whether to perform the initial TLS handshake before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. Requires version 4.60.0 or later. **Type**: `bool` **Default**: `false` ### [](#urls)`urls[]` A list of URLs to connect to. If a list item contains commas, it will be expanded into multiple URLs. **Type**: `array` ```yaml # Examples: urls: - "nats://127.0.0.1:4222" # --- urls: - "nats://username:password@127.0.0.1:4222" ``` --- # Page 258: noop **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/noop.md --- # noop --- title: noop latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/noop page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/noop.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/noop.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- **Type:** Processor ▼ [Processor](/redpanda-connect/components/processors/noop/)[Cache](/redpanda-connect/components/caches/noop/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/noop/ "View the Cloud version of this component"), Self-Managed Noop is a processor that does nothing, the message passes through unchanged. Why? Sometimes doing nothing is the braver option. ```yml # Config fields, showing default values label: "" noop: {} ``` --- # Page 259: ollama_chat **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/ollama_chat.md --- # ollama\_chat --- title: ollama_chat latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/ollama_chat page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/ollama_chat.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/ollama_chat.adoc page-git-created-date: "2024-08-15" page-git-modified-date: "2025-10-06" --- **Available in:** Self-Managed Generates responses to messages in a chat conversation using the Ollama API and external tools. Introduced in version 4.32.0. #### Common ```yml processors: label: "" ollama_chat: model: "" # No default (required) prompt: "" # No default (optional) image: "" # No default (optional) response_format: text max_tokens: "" # No default (optional) temperature: "" # No default (optional) save_prompt_metadata: false history: "" # No default (optional) tools: [] runner: context_size: "" # No default (optional) batch_size: "" # No default (optional) gpu_layers: "" # No default (optional) threads: "" # No default (optional) use_mmap: "" # No default (optional) server_address: "" # No default (optional) ``` #### Advanced ```yml processors: label: "" ollama_chat: model: "" # No default (required) prompt: "" # No default (optional) system_prompt: "" # No default (optional) image: "" # No default (optional) response_format: text max_tokens: "" # No default (optional) temperature: "" # No default (optional) num_keep: "" # No default (optional) seed: "" # No default (optional) top_k: "" # No default (optional) top_p: "" # No default (optional) repeat_penalty: "" # No default (optional) presence_penalty: "" # No default (optional) frequency_penalty: "" # No default (optional) stop: [] # No default (optional) save_prompt_metadata: false history: "" # No default (optional) max_tool_calls: 3 tools: [] runner: context_size: "" # No default (optional) batch_size: "" # No default (optional) gpu_layers: "" # No default (optional) threads: "" # No default (optional) use_mmap: "" # No default (optional) server_address: "" # No default (optional) cache_directory: "" # No default (optional) download_url: "" # No default (optional) ``` This processor sends prompts to your chosen Ollama large language model (LLM) and generates text from the responses using the Ollama API and external tools. By default, the processor starts and runs a locally-installed Ollama server. Alternatively, to use an already running Ollama server, add your server details to the `server_address` field. You can [download and install Ollama from the Ollama website](https://ollama.com/download). For more information, see the [Ollama documentation](https://github.com/ollama/ollama/tree/main/docs) and [examples](#examples). ## [](#fields)Fields ### [](#cache_directory)`cache_directory` If `server_address` is not set - the directory to download the Ollama binary and use as a model cache. **Type**: `string` ```yaml # Examples: cache_directory: /opt/cache/connect/ollama ``` ### [](#download_url)`download_url` If `server_address` is not set - the URL to download the Ollama binary from. Defaults to the official Ollama GitHub release for this platform. **Type**: `string` ### [](#frequency_penalty)`frequency_penalty` Positive values penalize new tokens based on the frequency of their appearance in the text so far. This decreases the model’s likelihood to repeat the same line verbatim. **Type**: `float` ### [](#history)`history` Include historical messages in a chat request. You must use a Bloblang query to create an array of objects in the form of `[{"role": "", "content":""}]` where: - `role` is the sender of the original messages, either `system`, `user`, `assistant`, or `tool`. - `content` is the text of the original messages. **Type**: `string` ### [](#image)`image` An optional image to submit along with the [`prompt`](#prompt) value. The result is a byte array. Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: image: root = this.image.decode("base64") # decode base64 encoded image ``` ### [](#max_tokens)`max_tokens` The maximum number of tokens to predict and output. Limiting the amount of output means that requests are processed faster and have a fixed limit on the cost. **Type**: `int` ### [](#max_tool_calls)`max_tool_calls` The maximum number of sequential calls you can make to external tools to retrieve additional information to answer a prompt. **Type**: `int` **Default**: `3` ### [](#model)`model` The name of the Ollama LLM to use. For a full list of models, see the [Ollama website](https://ollama.com/models). **Type**: `string` ```yaml # Examples: model: llama3.1 # --- model: gemma2 # --- model: qwen2 # --- model: phi3 ``` ### [](#num_keep)`num_keep` Specify the number of tokens from the initial prompt to retain when the model resets its internal context. By default, this value is set to `4`. Use `-1` to retain all tokens from the initial prompt. **Type**: `int` ### [](#presence_penalty)`presence_penalty` Positive values penalize new tokens if they have appeared in the text so far. This increases the model’s likelihood to talk about new topics. **Type**: `float` ### [](#prompt)`prompt` The prompt you want to generate a response for. By default, the processor submits the entire payload as a string. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#repeat_penalty)`repeat_penalty` Sets how strongly to penalize repetitions. A higher value, for example 1.5, will penalize repetitions more strongly. A lower value, for example 0.9, will be more lenient. **Type**: `float` ### [](#response_format)`response_format` The format of the response the Ollama model generates. If specifying JSON output, then the `prompt` should specify that the output should be in JSON as well. **Type**: `string` **Default**: `text` **Options**: `text`, `json` ### [](#runner)`runner` Options for the model runner that are used when the model is first loaded into memory. **Type**: `object` ### [](#runner-batch_size)`runner.batch_size` The maximum number of requests to process in parallel. **Type**: `int` ### [](#runner-context_size)`runner.context_size` Sets the size of the context window used to generate the next token. Using a larger context window uses more memory and takes longer to process. **Type**: `int` ### [](#runner-gpu_layers)`runner.gpu_layers` This option allows offloading some layers to the GPU for computation. This generally results in increased performance. By default, the runtime decides the number of layers dynamically. **Type**: `int` ### [](#runner-threads)`runner.threads` Set the number of threads to use during generation. For optimal performance, it is recommended to set this value to the number of physical CPU cores your system has. By default, the runtime decides the optimal number of threads. **Type**: `int` ### [](#runner-use_mmap)`runner.use_mmap` Map the model into memory. This is only support on unix systems and allows loading only the necessary parts of the model as needed. **Type**: `bool` ### [](#save_prompt_metadata)`save_prompt_metadata` Set to `true` to save the prompt value to a metadata field (`@prompt`) on the corresponding output message. If you use the `system_prompt` field, its value is also saved to an `@system_prompt` metadata field on each output message. **Type**: `bool` **Default**: `false` ### [](#seed)`seed` Sets the random number seed to use for generation. Setting this to a specific number will make the model generate the same text for the same prompt. **Type**: `int` ```yaml # Examples: seed: 42 ``` ### [](#server_address)`server_address` The address of the Ollama server to use. Leave the field blank and the processor starts and runs a local Ollama server or specify the address of your own local or remote server. **Type**: `string` ```yaml # Examples: server_address: http://127.0.0.1:11434 ``` ### [](#stop)`stop[]` Sets the stop sequences to use. When this pattern is encountered, the LLM stops generating text and returns the final response. **Type**: `array` ### [](#system_prompt)`system_prompt` The system prompt to submit to the Ollama LLM. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#temperature)`temperature` The temperature of the model. Increasing the temperature makes the model answer more creatively. **Type**: `int` ### [](#tools)`tools[]` The external tools the LLM can invoke, such as functions, APIs, or web browsing. You can build a series of processors that include definitions of these tools, and the specified LLM can choose when to invoke them to help answer a prompt. For more information, see [examples](#examples). **Type**: `object` **Default**: `[]` ### [](#tools-description)`tools[].description` A description of this tool, the LLM uses this to decide if the tool should be used. **Type**: `string` ### [](#tools-name)`tools[].name` The name of this tool. **Type**: `string` ### [](#tools-parameters)`tools[].parameters` The parameters the LLM needs to provide to invoke this tool. **Type**: `object` ### [](#tools-parameters-properties)`tools[].parameters.properties` The properties for the processor’s input data **Type**: `object` ### [](#tools-parameters-properties-description)`tools[].parameters.properties.description` A description of this parameter. **Type**: `string` ### [](#tools-parameters-properties-enum)`tools[].parameters.properties.enum[]` Specifies that this parameter is an enum and only these specific values should be used. **Type**: `array` **Default**: `[]` ### [](#tools-parameters-properties-type)`tools[].parameters.properties.type` The type of this parameter. **Type**: `string` ### [](#tools-parameters-required)`tools[].parameters.required[]` The required parameters for this pipeline. **Type**: `array` **Default**: `[]` ### [](#tools-processors)`tools[].processors[]` The pipeline to execute when the LLM uses this tool. **Type**: `processor` ### [](#top_k)`top_k` Reduces the probability of generating nonsense. A higher value, for example `100`, will give more diverse answers. A lower value, for example `10`, will be more conservative. **Type**: `int` ### [](#top_p)`top_p` Works together with `top-k`. A higher value, for example 0.95, will lead to more diverse text. A lower value, for example 0.5, will generate more focused and conservative text. **Type**: `float` ## [](#examples)Examples ### [](#use-llava-to-analyze-an-image)Use Llava to analyze an image This example fetches image URLs from stdin and has a multimodal LLM describe the image. ```yaml input: stdin: scanner: lines: {} pipeline: processors: - http: verb: GET url: "${!content().string()}" - ollama_chat: model: llava prompt: "Describe the following image" image: "root = content()" output: stdout: codec: lines ``` ### [](#use-subpipelines-as-tool-calls)Use subpipelines as tool calls This example allows llama3.2 to execute a subpipeline as a tool call to get more data. ```yaml input: generate: count: 1 mapping: | root = "What is the weather like in Chicago?" pipeline: processors: - ollama_chat: model: llama3.2 prompt: "${!content().string()}" tools: - name: GetWeather description: "Retrieve the weather for a specific city" parameters: required: ["city"] properties: city: type: string description: the city to lookup the weather for processors: - http: verb: GET url: 'https://wttr.in/${!this.city}?T' headers: # Spoof curl user-ageent to get a plaintext text User-Agent: curl/8.11.1 output: stdout: {} ``` --- # Page 260: ollama_embeddings **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/ollama_embeddings.md --- # ollama\_embeddings --- title: ollama_embeddings latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/ollama_embeddings page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/ollama_embeddings.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/ollama_embeddings.adoc page-git-created-date: "2024-08-15" page-git-modified-date: "2025-10-06" --- **Available in:** Self-Managed Generates vector embeddings from text, using the Ollama API. Introduced in version 4.32.0. #### Common ```yml processors: label: "" ollama_embeddings: model: "" # No default (required) text: "" # No default (optional) runner: context_size: "" # No default (optional) batch_size: "" # No default (optional) gpu_layers: "" # No default (optional) threads: "" # No default (optional) use_mmap: "" # No default (optional) server_address: "" # No default (optional) ``` #### Advanced ```yml processors: label: "" ollama_embeddings: model: "" # No default (required) text: "" # No default (optional) runner: context_size: "" # No default (optional) batch_size: "" # No default (optional) gpu_layers: "" # No default (optional) threads: "" # No default (optional) use_mmap: "" # No default (optional) server_address: "" # No default (optional) cache_directory: "" # No default (optional) download_url: "" # No default (optional) ``` This processor sends text to your chosen Ollama large language model (LLM) and creates vector embeddings, using the Ollama API. Vector embeddings are long arrays of numbers that represent values or objects, in this case text. By default, the processor starts and runs a locally installed Ollama server. Alternatively, to use an already running Ollama server, add your server details to the `server_address` field. You can [download and install Ollama from the Ollama website](https://ollama.com/download). For more information, see the [Ollama documentation](https://github.com/ollama/ollama/tree/main/docs). ## [](#fields)Fields ### [](#cache_directory)`cache_directory` If `server_address` is not set - the directory to download the ollama binary and use as a model cache. **Type**: `string` ```yaml # Examples: cache_directory: /opt/cache/connect/ollama ``` ### [](#download_url)`download_url` If `server_address` is not set - the URL to download the ollama binary from. Defaults to the official Ollama GitHub release for this platform. **Type**: `string` ### [](#model)`model` The name of the Ollama LLM to use. For a full list of models, see the [Ollama website](https://ollama.com/models). **Type**: `string` ```yaml # Examples: model: nomic-embed-text # --- model: mxbai-embed-large # --- model: snowflake-artic-embed # --- model: all-minilm ``` ### [](#runner)`runner` Options for the model runner that are used when the model is first loaded into memory. **Type**: `object` ### [](#runner-batch_size)`runner.batch_size` The maximum number of requests to process in parallel. **Type**: `int` ### [](#runner-context_size)`runner.context_size` Sets the size of the context window used to generate the next token. Using a larger context window uses more memory and takes longer to processor. **Type**: `int` ### [](#runner-gpu_layers)`runner.gpu_layers` This option allows offloading some layers to the GPU for computation. This generally results in increased performance. By default, the runtime decides the number of layers dynamically. **Type**: `int` ### [](#runner-threads)`runner.threads` Set the number of threads to use during generation. For optimal performance, it is recommended to set this value to the number of physical CPU cores your system has. By default, the runtime decides the optimal number of threads. **Type**: `int` ### [](#runner-use_mmap)`runner.use_mmap` Map the model into memory. This is only support on unix systems and allows loading only the necessary parts of the model as needed. **Type**: `bool` ### [](#server_address)`server_address` The address of the Ollama server to use. Leave the field blank and the processor starts and runs a local Ollama server or specify the address of your own local or remote server. **Type**: `string` ```yaml # Examples: server_address: http://127.0.0.1:11434 ``` ### [](#text)`text` The text you want to create vector embeddings for. By default, the processor submits the entire payload as a string. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 261: ollama_moderation **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/ollama_moderation.md --- # ollama\_moderation --- title: ollama_moderation latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/ollama_moderation page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/ollama_moderation.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/ollama_moderation.adoc page-git-created-date: "2025-01-28" page-git-modified-date: "2025-10-06" --- **Available in:** Self-Managed Generates responses to messages in a chat conversation using the Ollama API, and checks the responses to make sure they do not violate [safety or security standards](https://mlcommons.org/2024/04/mlc-aisafety-v0-5-poc/). Introduced in version 4.45.0. #### Common ```yml processors: label: "" ollama_moderation: model: "" # No default (required) prompt: "" # No default (required) response: "" # No default (required) runner: context_size: "" # No default (optional) batch_size: "" # No default (optional) gpu_layers: "" # No default (optional) threads: "" # No default (optional) use_mmap: "" # No default (optional) server_address: "" # No default (optional) ``` #### Advanced ```yml processors: label: "" ollama_moderation: model: "" # No default (required) prompt: "" # No default (required) response: "" # No default (required) runner: context_size: "" # No default (optional) batch_size: "" # No default (optional) gpu_layers: "" # No default (optional) threads: "" # No default (optional) use_mmap: "" # No default (optional) server_address: "" # No default (optional) cache_directory: "" # No default (optional) download_url: "" # No default (optional) ``` This processor checks the safety of responses from your chosen large language model (LLM) using either [Llama Guard 3](https://ollama.com/library/llama-guard3) or [ShieldGemma](https://ollama.com/library/shieldgemma). By default, the processor starts and runs a locally-installed Ollama server. Alternatively, to use an already running Ollama server, add your server details to the `server_address` field. You can [download and install Ollama from the Ollama website](https://ollama.com/download). For more information, see the [Ollama documentation](https://github.com/ollama/ollama/tree/main/docs) and [Examples](#examples). To check the safety of your prompts, see the [`ollama_chat` processor](../ollama_chat/#examples) documentation. ## [](#fields)Fields ### [](#cache_directory)`cache_directory` If the `server_address` is not set, download the Ollama binary to this directory and use it as a model cache. **Type**: `string` ```yaml # Examples: cache_directory: /opt/cache/connect/ollama ``` ### [](#download_url)`download_url` If `server_address` is not set, download the Ollama binary from this URL. The default value is the official Ollama GitHub release for this platform. **Type**: `string` ### [](#model)`model` The name of the Ollama LLM to use. **Type**: `string` | Option | Summary | | --- | --- | | llama-guard3 | When using llama-guard3, two pieces of metadata is added: @safe with the value of yes or no and the second being @category for the safety category violation. For more information see the Llama Guard 3 Model Card. | | shieldgemma | When using shieldgemma, the model output is a single piece of metadata of @safe with a value of yes or no if the response is not in violation of its defined safety policies. | ```yaml # Examples: model: llama-guard3 # --- model: shieldgemma ``` ### [](#prompt)`prompt` The prompt you used to generate a response from an LLM. If you’re using the `ollama_chat` processor, you can set the `save_prompt_metadata` field to save the contents of your prompts. You can then run them through `ollama_moderation` processor to check the model responses for safety. For more details, see [Examples](#examples). You can also check the safety of your prompts. For more information, see the [`ollama_chat` processor](../ollama_chat/#examples) documentation. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#response)`response` The LLM’s response that you want to check for safety. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#runner)`runner` Options for the model runner that are used when the model is first loaded into memory. **Type**: `object` ### [](#runner-batch_size)`runner.batch_size` The maximum number of requests to process in parallel. **Type**: `int` ### [](#runner-context_size)`runner.context_size` Sets the size of the context window used to generate the next token. Using a larger context window uses more memory and takes longer to process. **Type**: `int` ### [](#runner-gpu_layers)`runner.gpu_layers` Sets the number of layers to offload to the GPU for computation. This generally results in increased performance. By default, the runtime decides the number of layers dynamically. **Type**: `int` ### [](#runner-threads)`runner.threads` Sets the number of threads to use during response generation. For optimal performance, set this value to the number of physical CPU cores your system has. By default, the runtime decides the optimal number of threads. **Type**: `int` ### [](#runner-use_mmap)`runner.use_mmap` Map the model into memory. Set to `true` to load only the necessary parts of the model into memory. This setting is only supported on Unix systems. **Type**: `bool` ### [](#server_address)`server_address` The address of the Ollama server to use. Leave this field blank and the processor starts and runs a local Ollama server, or specify the address of your own local or remote server. **Type**: `string` ```yaml # Examples: server_address: http://127.0.0.1:11434 ``` ## [](#examples)Examples ### [](#use-llama-guard-3-classify-a-llm-response)Use Llama Guard 3 classify a LLM response This example uses Llama Guard 3 to check if another model responded with a safe or unsafe content. ```yaml input: stdin: scanner: lines: {} pipeline: processors: - ollama_chat: model: llava prompt: "${!content().string()}" save_prompt_metadata: true - ollama_moderation: model: llama-guard3 prompt: "${!@prompt}" response: "${!content().string()}" - mapping: | root.response = content().string() root.is_safe = @safe output: stdout: codec: lines ``` --- # Page 262: openai_chat_completion **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/openai_chat_completion.md --- # openai\_chat\_completion --- title: openai_chat_completion latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/openai_chat_completion page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/openai_chat_completion.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/openai_chat_completion.adoc page-git-created-date: "2024-08-15" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/openai_chat_completion/ "View the Cloud version of this component"), Self-Managed Generates responses to messages in a chat conversation, using the OpenAI API and external tools. Introduced in version 4.32.0. #### Common ```yml processors: label: "" openai_chat_completion: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) prompt: "" # No default (optional) system_prompt: "" # No default (optional) history: "" # No default (optional) image: "" # No default (optional) max_tokens: "" # No default (optional) temperature: "" # No default (optional) user: "" # No default (optional) response_format: text json_schema: name: "" # No default (required) description: "" # No default (optional) schema: "" # No default (required) tools: [] # No default (required) ``` #### Advanced ```yml processors: label: "" openai_chat_completion: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) prompt: "" # No default (optional) system_prompt: "" # No default (optional) history: "" # No default (optional) image: "" # No default (optional) max_tokens: "" # No default (optional) temperature: "" # No default (optional) user: "" # No default (optional) response_format: text json_schema: name: "" # No default (required) description: "" # No default (optional) schema: "" # No default (required) schema_registry: url: "" # No default (required) name_prefix: schema_registry_id_ subject: "" # No default (required) refresh_interval: "" # No default (optional) tls: skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} top_p: "" # No default (optional) frequency_penalty: "" # No default (optional) presence_penalty: "" # No default (optional) seed: "" # No default (optional) stop: [] # No default (optional) tools: [] # No default (required) ``` This processor sends user prompts to the OpenAI API, and the specified large language model (LLM) generates responses using all available context, including supplementary data provided by [external tools](#tools). By default, the processor submits the entire payload of each message as a string, unless you use the `prompt` configuration field to customize it. To learn more about chat completion, see the [OpenAI API documentation](https://platform.openai.com/docs/guides/chat-completions), and [Examples](#Examples). ## [](#fields)Fields ### [](#api_key)`api_key` The API secret key for OpenAI API. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#frequency_penalty)`frequency_penalty` Specify a number between `-2.0` and `2.0`. Positive values penalize new tokens based on the frequency of their appearance in the text so far. This decreases the model’s likelihood to repeat the same line verbatim. **Type**: `float` ### [](#history)`history` Include messages from a prior conversation. You must use a Bloblang query to create an array of objects in the form of `[{"role": "user", "content": ""}, {"role":"assistant", "content":""}]` where: - `role` is the sender of the original messages, either `system`, `user`, or `assistant`. - `content` is the text of the original messages. For more information, see [Examples](#Examples). **Type**: `string` ### [](#image)`image` An optional image to submit along with the prompt. The result of the Bloblang mapping must be a byte array. Requires version 4.38.0 or later. **Type**: `string` ```yaml # Examples: image: root = this.image.decode("base64") # decode base64 encoded image ``` ### [](#json_schema)`json_schema` The JSON schema used by the model when generating responses in `json_schema` format. To learn more about supported JSON schema features, see the [OpenAI documentation](https://platform.openai.com/docs/guides/structured-outputs/supported-schemas). **Type**: `object` ### [](#json_schema-description)`json_schema.description` An optional description, which helps the model understand the schema’s purpose. **Type**: `string` ### [](#json_schema-name)`json_schema.name` The name of the JSON schema to use. **Type**: `string` ### [](#json_schema-schema)`json_schema.schema` The JSON schema for the model to use when generating the output. **Type**: `string` ### [](#max_tokens)`max_tokens` The maximum number of tokens to generate for chat completion. **Type**: `int` ### [](#model)`model` The name of the OpenAI model to use. **Type**: `string` ```yaml # Examples: model: gpt-4o # --- model: gpt-4o-mini # --- model: gpt-4 # --- model: gpt4-turbo ``` ### [](#presence_penalty)`presence_penalty` Specify a number between `-2.0` and `2.0`. Positive values penalize new tokens if they have appeared in the text so far. This increases the model’s likelihood to talk about new topics. **Type**: `float` ### [](#prompt)`prompt` The user prompt for which a response is generated. By default, the processor sends the entire payload as a string unless customized using this field. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#response_format)`response_format` Specify the configured [model’s](#model) output format. If you choose the `json_schema` option, you must also configure a `json_schema` or `schema_registry`. **Type**: `string` **Default**: `text` **Options**: `text`, `json`, `json_schema` ### [](#schema_registry)`schema_registry` The schema registry to dynamically load schemas for model responses in `json_schema` format. Schemas must be in JSON format. To learn more about supported JSON schema features, see the [OpenAI documentation](https://platform.openai.com/docs/guides/structured-outputs/supported-schemas). **Type**: `object` ### [](#schema_registry-basic_auth)`schema_registry.basic_auth` Configure basic authentication for requests from this component to your schema registry. **Type**: `object` ### [](#schema_registry-basic_auth-enabled)`schema_registry.basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-basic_auth-password)`schema_registry.basic_auth.password` The password to use for authentication. Used together with `username` for basic authentication or with encrypted private keys for secure access. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-basic_auth-username)`schema_registry.basic_auth.username` The username of the account credentials to authenticate as. Used together with `password` for basic authentication. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt)`schema_registry.jwt` Beta Allows you to specify JWT authentication. **Type**: `object` ### [](#schema_registry-jwt-claims)`schema_registry.jwt.claims` Values used to pass the identity of the authenticated entity to the service provider. In this case, between this component and the schema registry. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-enabled)`schema_registry.jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-jwt-headers)`schema_registry.jwt.headers` The key/value pairs that identify the type of token and signing algorithm (optional). **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-private_key_file)`schema_registry.jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt-signing_method)`schema_registry.jwt.signing_method` The cryptographic algorithm used to sign the JWT token. Supported algorithms include RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#schema_registry-name_prefix)`schema_registry.name_prefix` A prefix to add to the schema registry name. To form the complete schema registry name, the schema ID is appended as a suffix. **Type**: `string` **Default**: `schema_registry_id_` ### [](#schema_registry-oauth)`schema_registry.oauth` Configure OAuth version 1.0 to give this component authorized access to your schema registry. **Type**: `object` ### [](#schema_registry-oauth-access_token)`schema_registry.oauth.access_token` The value this component can use to gain access to the data in the schema registry. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-access_token_secret)`schema_registry.oauth.access_token_secret` The secret that establishes ownership of the `oauth.access_token` in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_key)`schema_registry.oauth.consumer_key` The value used to identify this component or client to your schema registry. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_secret)`schema_registry.oauth.consumer_secret` The secret that establishes ownership of the consumer key in OAuth 1.0 authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-enabled)`schema_registry.oauth.enabled` Whether to enable OAuth version 1.0 authentication for requests to the schema registry. **Type**: `bool` **Default**: `false` ### [](#schema_registry-refresh_interval)`schema_registry.refresh_interval` How frequently to poll the schema registry for updates. If not specified, the schema does not refresh automatically. **Type**: `string` ### [](#schema_registry-subject)`schema_registry.subject` The subject name used to fetch the schema from the schema registry. **Type**: `string` ### [](#schema_registry-tls)`schema_registry.tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#schema_registry-tls-client_certs)`schema_registry.tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#schema_registry-tls-client_certs-cert)`schema_registry.tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-cert_file)`schema_registry.tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key)`schema_registry.tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key_file)`schema_registry.tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-password)`schema_registry.tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#schema_registry-tls-enable_renegotiation)`schema_registry.tls.enable_renegotiation` Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-root_cas)`schema_registry.tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#schema_registry-tls-root_cas_file)`schema_registry.tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#schema_registry-tls-skip_cert_verify)`schema_registry.tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#schema_registry-url)`schema_registry.url` The base URL of the schema registry service. **Type**: `string` ### [](#seed)`seed` When set to a specific number, Redpanda Connect attempts to generate consistent responses for requests that use the same prompt, seed, and parameters. **Type**: `int` ### [](#server_address)`server_address` The OpenAI API endpoint to which the processor sends requests. Update the default value to use a different OpenAI-compatible service. **Type**: `string` **Default**: `[https://api.openai.com/v1](https://api.openai.com/v1)` ### [](#stop)`stop[]` Specify up to four stop sequences to use. When the model encounters a stop pattern, it stops generating text and returns the final response. **Type**: `array` ### [](#system_prompt)`system_prompt` The system prompt to submit along with the user prompt. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#temperature)`temperature` Choose a sampling temperature between `0` and `2`: - Higher values, such as `0.8` make the output more random. - Lower values, such as `0.2` make the output more focused and deterministic. Redpanda recommends adding a value for this field or [`top_p`](#top_p), but not both. **Type**: `float` ### [](#tools)`tools[]` External tools the model can invoke, such as functions, APIs, or web browsing. You can build a series of processors that include definitions of these tools, and the specified model can choose when to invoke them to help answer a prompt. For more information, see [Examples](#Examples). > 📝 **NOTE** > > If you don’t want to use external tools, enter an empty array `tools:[]`. **Type**: `object` ### [](#tools-description)`tools[].description` A description of this tool, the LLM uses this to decide if the tool should be used. **Type**: `string` ### [](#tools-name)`tools[].name` The name of this tool. **Type**: `string` ### [](#tools-parameters)`tools[].parameters` The parameters the LLM needs to provide to invoke this tool. **Type**: `object` **Default**: `[]` ### [](#tools-parameters-properties)`tools[].parameters.properties` The properties for the processor’s input data **Type**: `object` ### [](#tools-parameters-properties-description)`tools[].parameters.properties.description` A description of this parameter. **Type**: `string` ### [](#tools-parameters-properties-enum)`tools[].parameters.properties.enum[]` Specifies that this parameter is an enum and only these specific values should be used. **Type**: `array` **Default**: `[]` ### [](#tools-parameters-properties-type)`tools[].parameters.properties.type` The type of this parameter. **Type**: `string` ### [](#tools-parameters-required)`tools[].parameters.required[]` The required parameters for this pipeline. **Type**: `array` **Default**: `[]` ### [](#tools-processors)`tools[].processors[]` The pipeline to execute when the LLM uses this tool. **Type**: `processor` ### [](#top_p)`top_p` An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with `top_p` probability mass. For example, a `top_p` of `0.1` means only the tokens comprising the top 10% probability mass are sampled. Redpanda recommends adding a value for this field or `temperature`, but not both. **Type**: `float` ### [](#user)`user` A unique identifier that represents the end-user generating the prompt. This value can help OpenAI monitor and detect [platform abuse](https://openai.com/policies/usage-policies/). This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` nclude::redpanda-connect:components:partial$examples/processors/openai\_chat\_completion.adoc\[\] --- # Page 263: openai_embeddings **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/openai_embeddings.md --- # openai\_embeddings --- title: openai_embeddings latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/openai_embeddings page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/openai_embeddings.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/openai_embeddings.adoc page-git-created-date: "2024-08-15" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/openai_embeddings/ "View the Cloud version of this component"), Self-Managed Generates vector embeddings to represent input text, using the OpenAI API. Introduced in version 4.32.0. ```yml # Config fields, showing default values label: "" openai_embeddings: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: text-embedding-3-large # No default (required) text_mapping: "" # No default (optional) ``` This processor sends text strings to the OpenAI API, which generates vector embeddings. By default, the processor submits the entire payload of each message as a string, unless you use the `text_mapping` configuration field to customize it. To learn more about vector embeddings, see the [OpenAI API documentation](https://platform.openai.com/docs/guides/embeddings). ## [](#fields)Fields ### [](#api_key)`api_key` The API key for OpenAI API. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#dimensions)`dimensions` The number of dimensions the resulting output embeddings should have. Only supported in `text-embedding-3` and later models. **Type**: `int` ### [](#model)`model` The name of the OpenAI model to use. **Type**: `string` ```yaml # Examples: model: text-embedding-3-large # --- model: text-embedding-3-small # --- model: text-embedding-ada-002 ``` ### [](#server_address)`server_address` The Open API endpoint that the processor sends requests to. Update the default value to use another OpenAI compatible service. **Type**: `string` **Default**: `[https://api.openai.com/v1](https://api.openai.com/v1)` ### [](#text_mapping)`text_mapping` The text you want to generate a vector embedding for. By default, the processor submits the entire payload as a string. **Type**: `string` --- # Page 264: openai_image_generation **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/openai_image_generation.md --- # openai\_image\_generation --- title: openai_image_generation latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/openai_image_generation page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/openai_image_generation.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/openai_image_generation.adoc page-git-created-date: "2024-08-15" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/openai_image_generation/ "View the Cloud version of this component"), Self-Managed Generates an image from a text description and other attributes, using OpenAI API. Introduced in version 4.32.0. #### Common ```yml processors: label: "" openai_image_generation: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) prompt: "" # No default (optional) ``` #### Advanced ```yml processors: label: "" openai_image_generation: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) prompt: "" # No default (optional) quality: "" # No default (optional) size: "" # No default (optional) style: "" # No default (optional) ``` This processor sends an image description and other attributes, such as image size and quality to the OpenAI API, which generates an image. By default, the processor submits the entire payload of each message as a string, unless you use the `prompt` configuration field to customize it. To learn more about image generation, see the [OpenAI API documentation](https://platform.openai.com/docs/guides/images). ## [](#fields)Fields ### [](#api_key)`api_key` The API key for OpenAI API. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#model)`model` The name of the OpenAI model to use. **Type**: `string` ```yaml # Examples: model: dall-e-3 # --- model: dall-e-2 ``` ### [](#prompt)`prompt` A text description of the image you want to generate. The `prompt` field accepts a maximum of 1000 characters for `dall-e-2` and 4000 characters for `dall-e-3`. **Type**: `string` ### [](#quality)`quality` The quality of the image to generate. Use `hd` to create images with finer details and greater consistency across the image. This parameter is only supported for `dall-e-3` models. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: quality: standard # --- quality: hd ``` ### [](#server_address)`server_address` The Open API endpoint that the processor sends requests to. Update the default value to use another OpenAI compatible service. **Type**: `string` **Default**: `[https://api.openai.com/v1](https://api.openai.com/v1)` ### [](#size)`size` The size of the generated image. Choose from `256x256`, `512x512`, or `1024x1024` for `dall-e-2`. Choose from `1024x1024`, `1792x1024`, or `1024x1792` for `dall-e-3` models. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: size: 1024x1024 # --- size: 512x512 # --- size: 1792x1024 # --- size: 1024x1792 ``` ### [](#style)`style` The style of the generated image. Choose from `vivid` or `natural`. Vivid causes the model to lean towards generating hyperreal and dramatic images. Natural causes the model to produce more natural, less hyperreal looking images. This parameter is only supported for `dall-e-3`. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: style: vivid # --- style: natural ``` --- # Page 265: openai_speech **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/openai_speech.md --- # openai\_speech --- title: openai_speech latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/openai_speech page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/openai_speech.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/openai_speech.adoc page-git-created-date: "2024-08-15" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/openai_speech/ "View the Cloud version of this component"), Self-Managed Generates audio from a text description and other attributes, using OpenAI API. Introduced in version 4.32.0. #### Common ```yml processors: label: "" openai_speech: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) input: "" # No default (optional) voice: "" # No default (required) ``` #### Advanced ```yml processors: label: "" openai_speech: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) input: "" # No default (optional) voice: "" # No default (required) response_format: "" # No default (optional) ``` This processor sends a text description and other attributes, such as a voice type and format to the OpenAI API, which generates audio. By default, the processor submits the entire payload of each message as a string, unless you use the `input` configuration field to customize it. To learn more about turning text into spoken audio, see the [OpenAI API documentation](https://platform.openai.com/docs/guides/text-to-speech). ## [](#fields)Fields ### [](#api_key)`api_key` The API key for OpenAI API. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#input)`input` A text description of the audio you want to generate. The `input` field accepts a maximum of 4096 characters. **Type**: `string` ### [](#model)`model` The name of the OpenAI model to use. **Type**: `string` ```yaml # Examples: model: tts-1 # --- model: tts-1-hd ``` ### [](#response_format)`response_format` The format to generate audio in. Default is `mp3`. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: response_format: mp3 # --- response_format: opus # --- response_format: aac # --- response_format: flac # --- response_format: wav # --- response_format: pcm ``` ### [](#server_address)`server_address` The Open API endpoint that the processor sends requests to. Update the default value to use another OpenAI compatible service. **Type**: `string` **Default**: `[https://api.openai.com/v1](https://api.openai.com/v1)` ### [](#voice)`voice` The type of voice to use when generating the audio. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: voice: alloy # --- voice: echo # --- voice: fable # --- voice: onyx # --- voice: nova # --- voice: shimmer ``` --- # Page 266: openai_transcription **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/openai_transcription.md --- # openai\_transcription --- title: openai_transcription latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/openai_transcription page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/openai_transcription.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/openai_transcription.adoc page-git-created-date: "2024-08-15" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/openai_transcription/ "View the Cloud version of this component"), Self-Managed Generates a transcription of spoken audio in the input language, using the OpenAI API. Introduced in version 4.32.0. #### Common ```yml processors: label: "" openai_transcription: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) file: "" # No default (required) ``` #### Advanced ```yml processors: label: "" openai_transcription: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) file: "" # No default (required) language: "" # No default (optional) prompt: "" # No default (optional) ``` This processor sends an audio file object along with the input language to OpenAI API to generate a transcription. By default, the processor submits the entire payload of each message as a string, unless you use the `file` configuration field to customize it. To learn more about audio transcription, see the: [OpenAI API documentation](https://platform.openai.com/docs/guides/speech-to-text). ## [](#fields)Fields ### [](#api_key)`api_key` The API key for OpenAI API. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#file)`file` The audio file object (not file name) to transcribe, in one of the following formats: `flac`, `mp3`, `mp4`, `mpeg`, `mpga`, `m4a`, `ogg`, `wav`, or `webm`. **Type**: `string` ### [](#language)`language` The language of the input audio. Supplying the input language in ISO-639-1 format improves accuracy and latency. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: language: en # --- language: fr # --- language: de # --- language: zh ``` ### [](#model)`model` The name of the OpenAI model to use. **Type**: `string` ```yaml # Examples: model: whisper-1 ``` ### [](#prompt)`prompt` Optional text to guide the model’s style or continue a previous audio segment. The prompt should match the audio language. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#server_address)`server_address` The Open API endpoint that the processor sends requests to. Update the default value to use another OpenAI compatible service. **Type**: `string` **Default**: `[https://api.openai.com/v1](https://api.openai.com/v1)` --- # Page 267: openai_translation **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/openai_translation.md --- # openai\_translation --- title: openai_translation latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/openai_translation page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/openai_translation.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/openai_translation.adoc page-git-created-date: "2024-08-15" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/openai_translation/ "View the Cloud version of this component"), Self-Managed Translates spoken audio into English, using the OpenAI API. Introduced in version 4.32.0. #### Common ```yml processors: label: "" openai_translation: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) file: "" # No default (optional) ``` #### Advanced ```yml processors: label: "" openai_translation: server_address: https://api.openai.com/v1 api_key: "" # No default (required) model: "" # No default (required) file: "" # No default (optional) prompt: "" # No default (optional) ``` This processor sends an audio file object to OpenAI API to generate a translation. By default, the processor submits the entire payload of each message as a string, unless you use the `file` configuration field to customize it. To learn more about translation, see the [OpenAI API documentation](https://platform.openai.com/docs/guides/speech-to-text). ## [](#fields)Fields ### [](#api_key)`api_key` The API key for OpenAI API. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#file)`file` The audio file object (not file name) to translate, in one of the following formats: `flac`, `mp3`, `mp4`, `mpeg`, `mpga`, `m4a`, `ogg`, `wav`, or `webm`. **Type**: `string` ### [](#model)`model` The name of the OpenAI model to use. **Type**: `string` ```yaml # Examples: model: whisper-1 ``` ### [](#prompt)`prompt` Optional text to guide the model’s style or continue a previous audio segment. The prompt should match the audio language. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#server_address)`server_address` The Open API endpoint that the processor sends requests to. Update the default value to use another OpenAI compatible service. **Type**: `string` **Default**: `[https://api.openai.com/v1](https://api.openai.com/v1)` --- # Page 268: parallel **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/parallel.md --- # parallel --- title: parallel latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/parallel page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/parallel.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/parallel.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/parallel/ "View the Cloud version of this component"), Self-Managed A processor that applies a list of child processors to messages of a batch as though they were each a batch of one message (similar to the [`for_each`](../for_each/) processor), but where each message is processed in parallel. ```yml # Config fields, showing default values label: "" parallel: cap: 0 processors: [] # No default (required) ``` The field `cap`, if greater than zero, caps the maximum number of parallel processing threads. The functionality of this processor depends on being applied across messages that are batched. You can find out more about batching in [Message Batching](../../../configuration/batching/). ## [](#fields)Fields ### [](#cap)`cap` The maximum number of messages to have processing at a given time. **Type**: `int` **Default**: `0` ### [](#processors)`processors[]` A list of child processors to apply. **Type**: `processor` --- # Page 269: parquet_decode **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/parquet_decode.md --- # parquet\_decode --- title: parquet_decode latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/parquet_decode page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/parquet_decode.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/parquet_decode.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/parquet_decode/ "View the Cloud version of this component"), Self-Managed Decodes [Parquet files](https://parquet.apache.org/docs/) into a batch of structured messages. Introduced in version 4.4.0. ```yml # Configuration fields, showing default values label: "" parquet_decode: handle_logical_types: v1 ``` ## [](#fields)Fields ### [](#handle_logical_types)`handle_logical_types` Set to `v2` to enable enhanced decoding of logical types, or keep the default value (`v1`) to ignore logical type metadata when decoding values. In Parquet format, logical types are represented using standard physical types along with metadata that provides additional context. For example, UUIDs are stored as a `FIXED_LEN_BYTE_ARRAY` physical type, but the schema metadata identifies them as UUIDs. By enabling `v2`, this processor uses the metadata descriptions of logical types to produce more meaningful values during decoding. > 📝 **NOTE** > > For backward compatibility, this field enables logical-type handling for the specified Parquet format version, and all earlier versions. When creating new pipelines, Redpanda recommends that you use the newest documented version. **Type**: `string` **Default**: `v1` | Option | Summary | | --- | --- | | v1 | No special handling of logical types | | v2 | TIMESTAMP - decodes as an RFC3339 string describing the time. If the isAdjustedToUTC flag is set to true in the parquet file, the time zone will be set to UTC. If it is set to false the time zone will be set to local time.UUID - decodes as a string, i.e. 00112233-4455-6677-8899-aabbccddeeff. | ```yaml # Examples: handle_logical_types: v2 ``` ## [](#examples)Examples ### [](#reading-parquet-files-from-aws-s3)Reading Parquet Files from AWS S3 In this example we consume files from AWS S3 as they’re written by listening onto an SQS queue for upload events. We make sure to use the `to_the_end` scanner which means files are read into memory in full, which then allows us to use a `parquet_decode` processor to expand each file into a batch of messages. Finally, we write the data out to local files as newline delimited JSON. ```yaml input: aws_s3: bucket: TODO prefix: foos/ scanner: to_the_end: {} sqs: url: TODO processors: - parquet_decode: {} output: file: codec: lines path: './foos/${! meta("s3_key") }.jsonl' ``` --- # Page 270: parquet_encode **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/parquet_encode.md --- # parquet\_encode --- title: parquet_encode latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/parquet_encode page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/parquet_encode.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/parquet_encode.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/parquet_encode/ "View the Cloud version of this component"), Self-Managed Encodes [Parquet files](https://parquet.apache.org/docs/) from a batch of structured messages. Introduced in version 4.4.0. #### Common ```yml processors: label: "" parquet_encode: schema: [] # No default (optional) schema_metadata: "" default_compression: uncompressed ``` #### Advanced ```yml processors: label: "" parquet_encode: schema: [] # No default (optional) schema_metadata: "" default_compression: uncompressed default_encoding: DELTA_LENGTH_BYTE_ARRAY ``` ## [](#fields)Fields ### [](#default_compression)`default_compression` The default compression type to use for fields. **Type**: `string` **Default**: `uncompressed` **Options**: `uncompressed`, `snappy`, `gzip`, `brotli`, `zstd`, `lz4raw` ### [](#default_encoding)`default_encoding` The default encoding type to use for fields. A custom default encoding is only necessary when consuming data with libraries that do not support `DELTA_LENGTH_BYTE_ARRAY`. Requires version 4.11.0 or later. **Type**: `string` **Default**: `DELTA_LENGTH_BYTE_ARRAY` **Options**: `DELTA_LENGTH_BYTE_ARRAY`, `PLAIN` ### [](#schema)`schema[]` Parquet schema. **Type**: `object` ### [](#schema-fields)`schema[].fields[]` A list of child fields. **Type**: `array` ```yaml # Examples: fields: - name: foo type: INT64 - name: bar type: BYTE_ARRAY ``` ### [](#schema-name)`schema[].name` The name of the column. **Type**: `string` ### [](#schema-optional)`schema[].optional` Whether the field is optional. **Type**: `bool` **Default**: `false` ### [](#schema-repeated)`schema[].repeated` Whether the field is repeated. **Type**: `bool` **Default**: `false` ### [](#schema-type)`schema[].type` The type of the column, only applicable for leaf columns with no child fields. Some logical types can be specified here such as UTF8. **Type**: `string` **Options**: `BOOLEAN`, `INT32`, `INT64`, `FLOAT`, `DOUBLE`, `BYTE_ARRAY`, `UTF8`, `TIMESTAMP`, `BSON`, `ENUM`, `JSON`, `UUID` ### [](#schema_metadata)`schema_metadata` Optionally specify a metadata field containing a schema definition to use for encoding instead of a statically defined schema. For batches of messages, the first message’s schema will be applied to all subsequent messages of the batch. **Type**: `string` **Default**: `""` ## [](#examples)Examples ### [](#writing-parquet-files-to-aws-s3)Writing Parquet Files to AWS S3 In this example we use the batching mechanism of an `aws_s3` output to collect a batch of messages in memory, which then converts it to a parquet file and uploads it. ```yaml output: aws_s3: bucket: TODO path: 'stuff/${! timestamp_unix() }-${! uuid_v4() }.parquet' batching: count: 1000 period: 10s processors: - parquet_encode: schema: - name: id type: INT64 - name: weight type: DOUBLE - name: content type: BYTE_ARRAY default_compression: zstd ``` --- # Page 271: parquet **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/parquet.md --- # parquet --- title: parquet latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/parquet page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/parquet.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/parquet.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Processor ▼ [Processor](/redpanda-connect/components/processors/parquet/)[Input](/redpanda-connect/components/inputs/parquet/) **Available in:** Self-Managed > ⚠️ **WARNING: Deprecated** > > Deprecated > > This component is deprecated and will be removed in the next major version release. Please consider moving onto [alternative components](#alternatives). Converts batches of documents to or from [Parquet files](https://parquet.apache.org/docs/). Introduced in version 3.62.0. ```yml # Config fields, showing default values label: "" parquet: operator: "" # No default (required) compression: snappy schema_file: schemas/foo.json # No default (optional) schema: |- # No default (optional) { "Tag": "name=root, repetitiontype=REQUIRED", "Fields": [ {"Tag":"name=name,inname=NameIn,type=BYTE_ARRAY,convertedtype=UTF8, repetitiontype=REQUIRED"}, {"Tag":"name=age,inname=Age,type=INT32,repetitiontype=REQUIRED"} ] } ``` ## [](#alternatives)Alternatives This processor is now deprecated, it’s recommended that you use the new [`parquet_decode`](../parquet_decode/) and [`parquet_encode`](../parquet_encode/) processors as they provide a number of advantages, the most important of which is better error messages for when schemas are mismatched or files could not be consumed. ## [](#troubleshooting)Troubleshooting This processor is experimental and the error messages that it provides are often vague and unhelpful. An error message of the form `interface \{} is nil, not ` implies that a field of the given type was expected but not found in the processed message when writing parquet files. Unfortunately the name of the field will sometimes be missing from the error, in which case it’s worth double checking the schema you provided to make sure that there are no typos in the field names, and if that doesn’t reveal the issue it can help to mark fields as OPTIONAL in the schema and gradually change them back to REQUIRED until the error returns. ## [](#define-the-schema)Define the schema The schema must be specified as a JSON string, containing an object that describes the fields expected at the root of each document. Each field can itself have more fields defined, allowing for nested structures: ```json { "Tag": "name=root, repetitiontype=REQUIRED", "Fields": [ {"Tag": "name=name, inname=NameIn, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"}, {"Tag": "name=age, inname=Age, type=INT32, repetitiontype=REQUIRED"}, {"Tag": "name=id, inname=Id, type=INT64, repetitiontype=REQUIRED"}, {"Tag": "name=weight, inname=Weight, type=FLOAT, repetitiontype=REQUIRED"}, { "Tag": "name=favPokemon, inname=FavPokemon, type=LIST, repetitiontype=OPTIONAL", "Fields": [ {"Tag": "name=name, inname=PokeName, type=BYTE_ARRAY, convertedtype=UTF8, repetitiontype=REQUIRED"}, {"Tag": "name=coolness, inname=Coolness, type=FLOAT, repetitiontype=REQUIRED"} ] } ] } ``` A schema can be derived from a source file using [https://github.com/xitongsys/parquet-go/tree/master/tool/parquet-tools](https://github.com/xitongsys/parquet-go/tree/master/tool/parquet-tools): ```sh ./parquet-tools -cmd schema -file foo.parquet ``` ## [](#fields)Fields ### [](#compression)`compression` The type of compression to use when writing parquet files, this field is ignored when consuming parquet files. **Type**: `string` **Default**: `snappy` **Options**: `uncompressed`, `snappy`, `gzip`, `lz4`, `zstd` ### [](#operator)`operator` Determines whether the processor converts messages into a parquet file or expands parquet files into messages. Converting into JSON allows subsequent processors and mappings to convert the data into any other format. **Type**: `string` | Option | Summary | | --- | --- | | from_json | Compress a batch of JSON documents into a file. | | to_json | Expand a file into one or more JSON messages. | ### [](#schema)`schema` A schema used to describe the parquet files being generated or consumed, the format of the schema is a JSON document detailing the tag and fields of documents. The schema can be found at: [https://pkg.go.dev/github.com/xitongsys/parquet-go#readme-json](https://pkg.go.dev/github.com/xitongsys/parquet-go#readme-json). Either a `schema_file` or `schema` field must be specified when creating Parquet files via the `from_json` operator. **Type**: `string` ```yaml # Examples: schema: |- { "Tag": "name=root, repetitiontype=REQUIRED", "Fields": [ {"Tag":"name=name,inname=NameIn,type=BYTE_ARRAY,convertedtype=UTF8, repetitiontype=REQUIRED"}, {"Tag":"name=age,inname=Age,type=INT32,repetitiontype=REQUIRED"} ] } ``` ### [](#schema_file)`schema_file` A file path containing a schema used to describe the parquet files being generated or consumed, the format of the schema is a JSON document detailing the tag and fields of documents. The schema can be found at: [https://pkg.go.dev/github.com/xitongsys/parquet-go#readme-json](https://pkg.go.dev/github.com/xitongsys/parquet-go#readme-json). Either a `schema_file` or `schema` field must be specified when creating Parquet files via the `from_json` operator. **Type**: `string` ```yaml # Examples: schema_file: schemas/foo.json ``` --- # Page 272: parse_log **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/parse_log.md --- # parse\_log --- title: parse_log latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/parse_log page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/parse_log.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/parse_log.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/parse_log/ "View the Cloud version of this component"), Self-Managed Parses common log [Formats](#formats) into [structured data](#codecs). This is easier and often much faster than [`grok`](../grok/). #### Common ```yml processors: label: "" parse_log: format: "" # No default (required) ``` #### Advanced ```yml processors: label: "" parse_log: format: "" # No default (required) best_effort: true allow_rfc3339: true default_year: current default_timezone: UTC ``` ## [](#fields)Fields ### [](#allow_rfc3339)`allow_rfc3339` Also accept timestamps in rfc3339 format while parsing. Applicable to format `syslog_rfc3164`. **Type**: `bool` **Default**: `true` ### [](#best_effort)`best_effort` Still returns partially parsed messages even if an error occurs. **Type**: `bool` **Default**: `true` ### [](#default_timezone)`default_timezone` Sets the strategy to decide the timezone for rfc3164 timestamps. Applicable to format `syslog_rfc3164`. This value should follow the [time.LoadLocation](https://golang.org/pkg/time/#LoadLocation) format. **Type**: `string` **Default**: `UTC` ### [](#default_year)`default_year` Sets the strategy used to set the year for rfc3164 timestamps. Applicable to format `syslog_rfc3164`. When set to `current` the current year will be set, when set to an integer that value will be used. Leave this field empty to not set a default year at all. **Type**: `string` **Default**: `current` ### [](#format)`format` A common log [format](#formats) to parse. **Type**: `string` **Options**: `syslog_rfc5424`, `syslog_rfc3164` ## [](#codecs)Codecs Currently the only supported structured data codec is `json`. ## [](#formats)Formats ### [](#syslog_rfc5424)`syslog_rfc5424` Attempts to parse a log following the [Syslog RFC5424](https://tools.ietf.org/html/rfc5424) spec. The resulting structured document may contain any of the following fields: - `message` (string) - `timestamp` (string, RFC3339) - `facility` (int) - `severity` (int) - `priority` (int) - `version` (int) - `hostname` (string) - `procid` (string) - `appname` (string) - `msgid` (string) - `structureddata` (object) ### [](#syslog_rfc3164)`syslog_rfc3164` Attempts to parse a log following the [Syslog rfc3164](https://tools.ietf.org/html/rfc3164) spec. The resulting structured document may contain any of the following fields: - `message` (string) - `timestamp` (string, RFC3339) - `facility` (int) - `severity` (int) - `priority` (int) - `hostname` (string) - `procid` (string) - `appname` (string) - `msgid` (string) --- # Page 273: processors **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/processors.md --- # processors --- title: processors latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/processors page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/processors.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/processors.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/processors/ "View the Cloud version of this component"), Self-Managed A processor grouping several sub-processors. ```yml # Config fields, showing default values label: "" processors: [] ``` This processor is useful in situations where you want to collect several processors under a single resource identifier, whether it is for making your configuration easier to read and navigate, or for improving the testability of your configuration. The behavior of child processors will match exactly the behavior they would have under any other processors block. ## [](#examples)Examples ### [](#grouped-processing)Grouped Processing Imagine we have a collection of processors who cover a specific functionality. We could use this processor to group them together and make it easier to read and mock during testing by giving the whole block a label: ```yaml pipeline: processors: - label: my_super_feature processors: - log: message: "Let's do something cool" - archive: format: json_array - mapping: root.items = this ``` --- # Page 274: protobuf **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/protobuf.md --- # protobuf --- title: protobuf latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/protobuf page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/protobuf.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/protobuf.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Available in:** Self-Managed Handles conversions between JSON documents and protobuf messages using reflection, which allows you to make conversions from or to the target `.proto` files. For more information about JSON mapping of protobuf messages, see [ProtoJSON Format](https://protobuf.dev/programming-guides/json/) and [Examples](#examples). ```yml # Configuration fields, showing default values label: "" protobuf: operator: "" # No default (required) message: "" # No default (required) discard_unknown: false use_proto_names: false import_paths: [] use_enum_numbers: false ``` ## [](#performance-considerations)Performance considerations Processing protobuf messages using reflection is less performant than using generated native code. For scenarios where performance is critical, consider using [Redpanda Connect plugins](https://github.com/benthosdev/benthos-plugin-example). ## [](#operators)Operators ### [](#to_json)`to_json` Converts protobuf messages into a generic JSON structure, which makes it easier to manipulate the contents of the JSON document within Redpanda Connect. ### [](#from_json)`from_json` Attempts to create a target protobuf message from a generic JSON structure. ## [](#fields)Fields ### [](#bsr)`bsr[]` Buf Schema Registry configuration. Either this field or `import_paths` must be populated. Note that this field is an array, and multiple BSR configurations can be provided. **Type**: `object` **Default**: `[]` ### [](#bsr-api_key)`bsr[].api_key` Buf Schema Registry server API key, can be left blank for a public registry. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#bsr-module)`bsr[].module` Module to fetch from a Buf Schema Registry e.g. 'buf.build/exampleco/mymodule'. **Type**: `string` ### [](#bsr-url)`bsr[].url` Buf Schema Registry URL, leave blank to extract from module. **Type**: `string` **Default**: `""` ### [](#bsr-version)`bsr[].version` Version to retrieve from the Buf Schema Registry, leave blank for latest. **Type**: `string` **Default**: `""` ### [](#discard_unknown)`discard_unknown` When set to `true`, the `from_json` operator discards fields that are unknown to the schema. **Type**: `bool` **Default**: `false` ### [](#import_paths)`import_paths[]` A list of directories that contain `.proto` files, including all definitions required for parsing the target message. If left empty, the current directory is used. This processor imports all `.proto` files listed within specified or default directories. **Type**: `array` **Default**: `[]` ### [](#message)`message` The fully-qualified name of the protobuf message to convert from or to JSON. **Type**: `string` ### [](#operator)`operator` The [operator](#operators) to execute. **Type**: `string` **Options**: `to_json`, `from_json`, `decode` ### [](#use_enum_numbers)`use_enum_numbers` When set to `true`, the `to_json` operator deserializes enumeration fields as their numerical values instead of their string names. For example, an enum field with a value of `ENUM_VALUE_ONE` is represented as `1` in the JSON output. **Type**: `bool` **Default**: `false` ### [](#use_proto_names)`use_proto_names` When set to `true`, the `to_json` operator deserializes fields exactly as named in schema file. **Type**: `bool` **Default**: `false` ## [](#examples)Examples ### [](#json-to-protobuf-using-schema-from-disk)JSON to Protobuf using Schema from Disk If we have the following protobuf definition within a directory called `testing/schema`: ```protobuf syntax = "proto3"; package testing; import "google/protobuf/timestamp.proto"; message Person { string first_name = 1; string last_name = 2; string full_name = 3; int32 age = 4; int32 id = 5; // Unique ID number for this person. string email = 6; google.protobuf.Timestamp last_updated = 7; } ``` And a stream of JSON documents of the form: ```json { "firstName": "caleb", "lastName": "quaye", "email": "caleb@myspace.com" } ``` We can convert the documents into protobuf messages with the following config: ```yaml pipeline: processors: - protobuf: operator: from_json message: testing.Person import_paths: [ testing/schema ] ``` ### [](#protobuf-to-json-using-schema-from-disk)Protobuf to JSON using Schema from Disk If we have the following protobuf definition within a directory called `testing/schema`: ```protobuf syntax = "proto3"; package testing; import "google/protobuf/timestamp.proto"; message Person { string first_name = 1; string last_name = 2; string full_name = 3; int32 age = 4; int32 id = 5; // Unique ID number for this person. string email = 6; google.protobuf.Timestamp last_updated = 7; } ``` And a stream of protobuf messages of the type `Person`, we could convert them into JSON documents of the format: ```json { "firstName": "caleb", "lastName": "quaye", "email": "caleb@myspace.com" } ``` With the following config: ```yaml pipeline: processors: - protobuf: operator: to_json message: testing.Person import_paths: [ testing/schema ] ``` ### [](#json-to-protobuf-using-buf-schema-registry)JSON to Protobuf using Buf Schema Registry If we have the following protobuf definition within a BSR module hosted at `buf.build/exampleco/mymodule`: ```protobuf syntax = "proto3"; package testing; import "google/protobuf/timestamp.proto"; message Person { string first_name = 1; string last_name = 2; string full_name = 3; int32 age = 4; int32 id = 5; // Unique ID number for this person. string email = 6; google.protobuf.Timestamp last_updated = 7; } ``` And a stream of JSON documents of the form: ```json { "firstName": "caleb", "lastName": "quaye", "email": "caleb@myspace.com" } ``` We can convert the documents into protobuf messages with the following config: ```yaml pipeline: processors: - protobuf: operator: from_json message: testing.Person bsr: - module: buf.build/exampleco/mymodule api_key: xxx ``` ### [](#protobuf-to-json-using-buf-schema-registry)Protobuf to JSON using Buf Schema Registry If we have the following protobuf definition within a BSR module hosted at `buf.build/exampleco/mymodule`: ```protobuf syntax = "proto3"; package testing; import "google/protobuf/timestamp.proto"; message Person { string first_name = 1; string last_name = 2; string full_name = 3; int32 age = 4; int32 id = 5; // Unique ID number for this person. string email = 6; google.protobuf.Timestamp last_updated = 7; } ``` And a stream of protobuf messages of the type `Person`, we could convert them into JSON documents of the format: ```json { "firstName": "caleb", "lastName": "quaye", "email": "caleb@myspace.com" } ``` With the following config: ```yaml pipeline: processors: - protobuf: operator: to_json message: testing.Person bsr: - module: buf.build/exampleco/mymodule api_key: xxxx ``` --- # Page 275: qdrant **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/qdrant.md --- # qdrant --- title: qdrant latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/qdrant page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/qdrant.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/qdrant.adoc page-git-created-date: "2025-05-19" page-git-modified-date: "2025-10-06" --- **Type:** Processor ▼ [Processor](/redpanda-connect/components/processors/qdrant/)[Output](/redpanda-connect/components/outputs/qdrant/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/qdrant/ "View the Cloud version of this component"), Self-Managed Query items within a [Qdrant collection](https://qdrant.tech/documentation/concepts/collections/) and filter the returned results. Introduced in version 4.54.0. #### Common ```yml processors: label: "" qdrant: grpc_host: "" # No default (required) api_token: "" collection_name: "" # No default (required) vector_mapping: "" # No default (required) filter: "" # No default (optional) payload_fields: [] payload_filter: include limit: 10 ``` #### Advanced ```yml processors: label: "" qdrant: grpc_host: "" # No default (required) api_token: "" tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] collection_name: "" # No default (required) vector_mapping: "" # No default (required) filter: "" # No default (optional) payload_fields: [] payload_filter: include limit: 10 ``` ## [](#fields)Fields ### [](#api_token)`api_token` The Qdrant API token to use for authentication, which defaults to an empty string. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#collection_name)`collection_name` The name of the Qdrant collection you want to query. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#filter)`filter` Specify additional filtering to perform on returned results. Mappings must return [a valid filter](https://qdrant.tech/documentation/concepts/filtering/) using the proto3-encoded form. **Type**: `string` ```yaml # Examples: filter: |- root.must = [ {"has_id":{"has_id":[{"num": 8}, { "uuid":"1234-5678-90ab-cdef" }]}}, {"field":{"key": "city", "match": {"text": "London"}}}, ] # --- filter: |- root.must = [ {"field":{"key": "city", "match": {"text": "London"}}}, ] root.must_not = [ {"field":{"color": "city", "match": {"text": "red"}}}, ] ``` ### [](#grpc_host)`grpc_host` The gRPC host of the Qdrant server. **Type**: `string` ```yaml # Examples: grpc_host: localhost:6334 # --- grpc_host: xyz-example.eu-central.aws.cloud.qdrant.io:6334 ``` ### [](#limit)`limit` The maximum number of points to return from the collection. **Type**: `int` **Default**: `10` ### [](#payload_fields)`payload_fields[]` The fields to include or exclude in returned results. Use this field in combination with `payload_filter`. **Type**: `array` **Default**: `[]` ### [](#payload_filter)`payload_filter` Whether to include or exclude the fields specified in `payload_fields` from the returned results. **Type**: `string` **Default**: `include` | Option | Summary | | --- | --- | | exclude | Exclude the payload fields specified in payload_fields. | | include | Include the payload fields specified in payload_fields. | ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#vector_mapping)`vector_mapping` A mapping to extract search vectors from the returned document. **Type**: `string` ```yaml # Examples: vector_mapping: root = [1.2, 0.5, 0.76] # --- vector_mapping: root = this.vector # --- vector_mapping: root = [[0.352,0.532,0.532,0.234],[0.352,0.532,0.532,0.234]] # --- vector_mapping: root = {"some_sparse": {"indices":[23,325,532],"values":[0.352,0.532,0.532]}} # --- vector_mapping: root = {"some_multi": [[0.352,0.532,0.532,0.234],[0.352,0.532,0.532,0.234]]} # --- vector_mapping: root = {"some_dense": [0.352,0.532,0.532,0.234]} ``` --- # Page 276: rate_limit **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/rate_limit.md --- # rate\_limit --- title: rate_limit latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/rate_limit page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/rate_limit.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/rate_limit.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/rate_limit/ "View the Cloud version of this component"), Self-Managed Throttles the throughput of a pipeline according to a specified [`rate_limit`](../../rate_limits/about/) resource. Rate limits are shared across components and therefore apply globally to all processing pipelines. ```yml # Config fields, showing default values label: "" rate_limit: resource: "" # No default (required) ``` ## [](#fields)Fields ### [](#resource)`resource` The target [`rate_limit` resource](../../rate_limits/about/). **Type**: `string` --- # Page 277: redis_script **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/redis_script.md --- # redis\_script --- title: redis_script latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/redis_script page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/redis_script.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/redis_script.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/redis_script/ "View the Cloud version of this component"), Self-Managed Performs actions against Redis using [LUA scripts](https://redis.io/docs/latest/develop/programmability/eval-intro/). Introduced in version 4.11.0. #### Common ```yml processors: label: "" redis_script: url: "" # No default (required) script: "" # No default (required) args_mapping: "" # No default (required) keys_mapping: "" # No default (required) ``` #### Advanced ```yml processors: label: "" redis_script: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] script: "" # No default (required) args_mapping: "" # No default (required) keys_mapping: "" # No default (required) retries: 3 retry_period: 500ms ``` Actions are performed for each message and the message contents are replaced with the result. In order to merge the result into the original message compose this processor within a [`branch` processor](../branch/). ## [](#examples)Examples ### [](#running-a-script)Running a script The following example will use a script execution to get next element from a sorted set and set its score with timestamp unix nano value. ```yaml pipeline: processors: - redis_script: url: TODO script: | local value = redis.call("ZRANGE", KEYS[1], '0', '0') if next(elements) == nil then return '' end redis.call("ZADD", "XX", KEYS[1], ARGV[1], value) return value keys_mapping: 'root = [ meta("key") ]' args_mapping: 'root = [ timestamp_unix_nano() ]' ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` A [Bloblang mapping](../../../guides/bloblang/about/) which should evaluate to an array of values matching in size to the number of arguments required for the specified Redis script. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.key ] # --- args_mapping: root = [ meta("kafka_key"), "hardcoded_value" ] ``` ### [](#client_name)`client_name` Set the client name for the Redis connection. Requires version 4.82.0 or later. **Type**: `string` **Default**: `redpanda-connect` ### [](#keys_mapping)`keys_mapping` A [Bloblang mapping](../../../guides/bloblang/about/) which should evaluate to an array of keys matching in size to the number of arguments required for the specified Redis script. **Type**: `string` ```yaml # Examples: keys_mapping: root = [ this.key ] # --- keys_mapping: root = [ meta("kafka_key"), this.count ] ``` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#retries)`retries` The maximum number of retries before abandoning a request. **Type**: `int` **Default**: `3` ### [](#retry_period)`retry_period` The time to wait before consecutive retry attempts. **Type**: `string` **Default**: `500ms` ### [](#script)`script` A script to use for the target operator. It has precedence over the 'command' field. **Type**: `string` ```yaml # Examples: script: return redis.call('set', KEYS[1], ARGV[1]) ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 278: redis **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/redis.md --- # redis --- title: redis latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/redis page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/redis.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/redis.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Processor ▼ [Processor](/redpanda-connect/components/processors/redis/)[Cache](/redpanda-connect/components/caches/redis/)[Rate\_limit](/redpanda-connect/components/rate_limits/redis/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/redis/ "View the Cloud version of this component"), Self-Managed Performs actions against Redis that aren’t possible using a [`cache`](../cache/) processor. Actions are performed for each message and the message contents are replaced with the result. In order to merge the result into the original message compose this processor within a [`branch` processor](../branch/). #### Common ```yml processors: label: "" redis: url: "" # No default (required) command: "" # No default (optional) args_mapping: "" # No default (optional) ``` #### Advanced ```yml processors: label: "" redis: url: "" # No default (required) kind: simple master: "" client_name: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] command: "" # No default (optional) args_mapping: "" # No default (optional) retries: 3 retry_period: 500ms ``` ## [](#examples)Examples ### [](#querying-cardinality)Querying Cardinality If given payloads containing a metadata field `set_key` it’s possible to query and store the cardinality of the set for each message using a [`branch` processor](../branch/) in order to augment rather than replace the message contents: ```yaml pipeline: processors: - branch: processors: - redis: url: TODO command: scard args_mapping: 'root = [ meta("set_key") ]' result_map: 'root.cardinality = this' ``` ### [](#running-total)Running Total If we have JSON data containing number of friends visited during covid 19: ```json {"name":"ash","month":"feb","year":2019,"friends_visited":10} {"name":"ash","month":"apr","year":2019,"friends_visited":-2} {"name":"bob","month":"feb","year":2019,"friends_visited":3} {"name":"bob","month":"apr","year":2019,"friends_visited":1} ``` We can add a field that contains the running total number of friends visited: ```json {"name":"ash","month":"feb","year":2019,"friends_visited":10,"total":10} {"name":"ash","month":"apr","year":2019,"friends_visited":-2,"total":8} {"name":"bob","month":"feb","year":2019,"friends_visited":3,"total":3} {"name":"bob","month":"apr","year":2019,"friends_visited":1,"total":4} ``` Using the `incrby` command: ```yaml pipeline: processors: - branch: processors: - redis: url: TODO command: incrby args_mapping: 'root = [ this.name, this.friends_visited ]' result_map: 'root.total = this' ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` A [Bloblang mapping](../../../guides/bloblang/about/) which should evaluate to an array of values matching in size to the number of arguments required for the specified Redis command. Requires version 4.3.0 or later. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.key ] # --- args_mapping: root = [ meta("kafka_key"), this.count ] ``` ### [](#client_name)`client_name` Set the client name for the Redis connection. Requires version 4.82.0 or later. **Type**: `string` **Default**: `redpanda-connect` ### [](#command)`command` The command to execute. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). Requires version 4.3.0 or later. **Type**: `string` ```yaml # Examples: command: scard # --- command: incrby # --- command: ${! meta("command") } ``` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `simple` **Options**: `simple`, `cluster`, `failover` ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yaml # Examples: master: mymaster ``` ### [](#retries)`retries` The maximum number of retries before abandoning a request. **Type**: `int` **Default**: `3` ### [](#retry_period)`retry_period` The time to wait before consecutive retry attempts. **Type**: `string` **Default**: `500ms` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yaml # Examples: url: redis://:6379 # --- url: redis://localhost:6379 # --- url: redis://foousername:foopassword@redisplace:6379 # --- url: redis://:foopassword@redisplace:6379 # --- url: redis://localhost:6379/1 # --- url: redis://localhost:6379/1,redis://localhost:6380/1 ``` --- # Page 279: redpanda_data_transform **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/redpanda_data_transform.md --- # redpanda\_data\_transform --- title: redpanda_data_transform latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/redpanda_data_transform page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/redpanda_data_transform.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/redpanda_data_transform.adoc page-git-created-date: "2024-08-15" page-git-modified-date: "2025-10-06" --- **Available in:** Self-Managed Executes a Redpanda Data Transform as a processor Introduced in version 4.31.0. #### Common ```yml processors: label: "" redpanda_data_transform: module_path: "" # No default (required) input_key: "" # No default (optional) output_key: "" # No default (optional) input_headers: include_prefixes: [] include_patterns: [] output_metadata: include_prefixes: [] include_patterns: [] ``` #### Advanced ```yml processors: label: "" redpanda_data_transform: module_path: "" # No default (required) input_key: "" # No default (optional) output_key: "" # No default (optional) input_headers: include_prefixes: [] include_patterns: [] output_metadata: include_prefixes: [] include_patterns: [] timestamp: "" # No default (optional) timeout: 10s max_memory_pages: 1600 ``` This processor executes a Redpanda Data Transform WebAssembly module, calling OnRecordWritten for each message being processed. You can find out about how transforms work here: [https://docs.redpanda.com/current/develop/data-transforms/how-transforms-work/](https://docs.redpanda.com/current/develop/data-transforms/how-transforms-work/) ## [](#fields)Fields ### [](#input_headers)`input_headers` Determine which (if any) metadata values should be added to messages as headers. **Type**: `object` ### [](#input_headers-include_patterns)`input_headers.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#input_headers-include_prefixes)`input_headers.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#input_key)`input_key` An optional key to populate for each message. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#max_memory_pages)`max_memory_pages` The maximum amount of wasm memory pages (64KiB) that an individual wasm module instance can use **Type**: `int` **Default**: `1600` ### [](#module_path)`module_path` The path of the target WASM module to execute. **Type**: `string` ### [](#output_key)`output_key` An optional name of metadata for an output message key. **Type**: `string` ### [](#output_metadata)`output_metadata` Determine which (if any) message headers should be added to the output as metadata. **Type**: `object` ### [](#output_metadata-include_patterns)`output_metadata.include_patterns[]` Provide a list of explicit metadata key regular expression (re2) patterns to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_patterns: - .* # --- include_patterns: - _timestamp_unix$ ``` ### [](#output_metadata-include_prefixes)`output_metadata.include_prefixes[]` Provide a list of explicit metadata key prefixes to match against. **Type**: `array` **Default**: `[]` ```yaml # Examples: include_prefixes: - foo_ - bar_ # --- include_prefixes: - kafka_ # --- include_prefixes: - content- ``` ### [](#timeout)`timeout` The maximum period of time for a message to be processed **Type**: `string` **Default**: `10s` ### [](#timestamp)`timestamp` An optional timestamp to set for each message. When left empty, the current timestamp is used. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: timestamp: ${! timestamp_unix() } # --- timestamp: ${! metadata("kafka_timestamp_ms") } ``` --- # Page 280: resource **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/resource.md --- # resource --- title: resource latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/resource page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/resource.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/resource.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-09" --- **Type:** Processor ▼ [Processor](/redpanda-connect/components/processors/resource/)[Input](/redpanda-connect/components/inputs/resource/)[Output](/redpanda-connect/components/outputs/resource/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/resource/ "View the Cloud version of this component"), Self-Managed Resource is a processor type that runs a processor resource identified by its label. ```yml # Config fields, showing default values resource: "" ``` This processor allows you to reference the same configured processor resource in multiple places, and can also tidy up large nested configs. For example, the config: ```yaml pipeline: processors: - mapping: | root.message = this root.meta.link_count = this.links.length() root.user.age = this.user.age.number() ``` Is equivalent to: ```yaml pipeline: processors: - resource: foo_proc processor_resources: - label: foo_proc mapping: | root.message = this root.meta.link_count = this.links.length() root.user.age = this.user.age.number() ``` You can find out more about resources in [Resources](../../../configuration/resources/) --- # Page 281: retry **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/retry.md --- # retry --- title: retry latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/retry page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/retry.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/retry.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Type:** Processor ▼ [Processor](/redpanda-connect/components/processors/retry/)[Output](/redpanda-connect/components/outputs/retry/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/retry/ "View the Cloud version of this component"), Self-Managed Attempts to execute a series of child processors until success. Introduced in version 4.27.0. ```yml # Config fields, showing default values label: "" retry: backoff: initial_interval: 500ms max_interval: 10s max_elapsed_time: 1m processors: [] # No default (required) parallel: false max_retries: 0 ``` Executes child processors and if a resulting message is errored then, after a specified backoff period, the same original message will be attempted again through those same processors. If the child processors result in more than one message then the retry mechanism will kick in if _any_ of the resulting messages are errored. It is important to note that any mutations performed on the message during these child processors will be discarded for the next retry, and therefore it is safe to assume that each execution of the child processors will always be performed on the data as it was when it first reached the retry processor. By default the retry backoff has a specified [`max_elapsed_time`](#backoffmax_elapsed_time), if this time period is reached during retries and an error still occurs these errored messages will proceed through to the next processor after the retry (or your outputs). Normal [error handling patterns](../../../configuration/error_handling/) can be used on these messages. In order to avoid permanent loops any error associated with messages as they first enter a retry processor will be cleared. ## [](#metadata)Metadata This processor adds the following metadata fields to each message: ```text - retry_count - The number of retry attempts. - backoff_duration - The total time elapsed while performing retries. ``` > ⚠️ **CAUTION: Batching** > > Batching > > If you wish to wrap a batch-aware series of processors then take a look at the [batching section](#batching). ## [](#examples)Examples ### [](#stop-ignoring-me-taz)Stop ignoring me Taz Here we have a config where I generate animal noises and send them to Taz via HTTP. Taz has a tendency to stop his servers whenever I dispatch my animals upon him, and therefore these HTTP requests sometimes fail. However, I have the retry processor and with this super power I can specify a back off policy and it will ensure that for each animal noise the HTTP processor is attempted until either it succeeds or my Redpanda Connect instance is stopped. I even go as far as to zero-out the maximum elapsed time field, which means that for each animal noise I will wait indefinitely, because I really really want Taz to receive every single animal noise that he is entitled to. ```yaml input: generate: interval: 1s mapping: 'root.noise = [ "woof", "meow", "moo", "quack" ].index(random_int(min: 0, max: 3))' pipeline: processors: - retry: backoff: initial_interval: 100ms max_interval: 5s max_elapsed_time: 0s processors: - http: url: 'http://example.com/try/not/to/dox/taz' verb: POST output: # Drop everything because it's junk data, I don't want it lol drop: {} ``` ## [](#fields)Fields ### [](#backoff)`backoff` Determine time intervals and cut offs for retry attempts. **Type**: `object` ### [](#backoff-initial_interval)`backoff.initial_interval` The initial period to wait between retry attempts. The retry interval increases for each failed attempt, up to the `backoff.max_interval` value. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `500ms` ```yaml # Examples: initial_interval: 50ms # --- initial_interval: 1s ``` ### [](#backoff-max_elapsed_time)`backoff.max_elapsed_time` The maximum overall period of time to spend on retry attempts before the request is aborted. Setting this value to a zeroed duration (such as `0s`) will result in unbounded retries. **Type**: `string` **Default**: `1m` ```yaml # Examples: max_elapsed_time: 1m # --- max_elapsed_time: 1h ``` ### [](#backoff-max_interval)`backoff.max_interval` The maximum period to wait between retry attempts **Type**: `string` **Default**: `10s` ```yaml # Examples: max_interval: 5s # --- max_interval: 1m ``` ### [](#max_retries)`max_retries` The maximum number of retry attempts before the request is aborted. Setting this value to `0` will result in unbounded number of retries. **Type**: `int` **Default**: `0` ### [](#parallel)`parallel` When processing batches of messages these batches are ignored and the processors apply to each message sequentially. However, when this field is set to `true` each message will be processed in parallel. Caution should be made to ensure that batch sizes do not surpass a point where this would cause resource (CPU, memory, API limits) contention. **Type**: `bool` **Default**: `false` ### [](#processors)`processors[]` A list of [processors](../about/) to execute on each message. **Type**: `processor` ## [](#batching)Batching When messages are batched the child processors of a retry are executed for each individual message in isolation, performed serially by default but in parallel when the field [`parallel`](#parallel) is set to `true`. This is an intentional limitation of the retry processor and is done in order to ensure that errors are correctly associated with a given input message. Otherwise, the archiving, expansion, grouping, filtering and so on of the child processors could obfuscate this relationship. If the target behavior of your retried processors is "batch aware", in that you wish to perform some processing across the entire batch of messages and repeat it in the event of errors, you can use an [`archive` processor](../archive/) to collapse the batch into an individual message. Then, within these child processors either perform your batch aware processing on the archive, or use an [`unarchive` processor](../unarchive/) in order to expand the single message back out into a batch. For example, if the retry processor were being used to wrap an HTTP request where the payload data is a batch archived into a JSON array it should look something like this: ```yaml pipeline: processors: - archive: format: json_array - retry: processors: - http: url: example.com/nope verb: POST - unarchive: format: json_array ``` --- # Page 282: salesforce **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/salesforce.md --- # salesforce --- title: salesforce latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/salesforce page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/salesforce.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/salesforce.adoc description: Fetches data from Salesforce based on input messages. page-git-created-date: "2026-03-31" page-git-modified-date: "2026-03-31" --- **Available in:** Self-Managed **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Fetches data from Salesforce based on input messages. Introduced in version 4.85.0. This processor takes input messages containing Salesforce queries and returns Salesforce data. Supports the following Salesforce resources: - sObject REST snapshot (all queryable sObjects or a filtered subset) - Filtered SOQL queries via REST or GraphQL - Change Data Capture (CDC) streaming via the Pub/Sub gRPC API - Platform Events via the Pub/Sub gRPC API #### Common ```yml processors: label: "" salesforce: org_url: "" # No default (required) client_id: "" # No default (required) client_secret: "" # No default (required) restapi_version: v65.0 request_timeout: 30s max_retries: 10 query_type: rest query: "" cdc_enabled: false cdc_objects: [] cdc_batch_size: 100 cdc_buffer_size: 1000 cdc_replay_preset: latest pubsub_topic: "" grpc_reconnect_base_delay: 500ms grpc_reconnect_max_delay: 30s grpc_reconnect_max_attempts: 0 grpc_shutdown_timeout: 10s cache_resource: salesforce_checkpoint parallel_fetch: 1 query_batch_size: 2000 rest_objects: [] ``` #### Advanced ```yml processors: label: "" salesforce: org_url: "" # No default (required) client_id: "" # No default (required) client_secret: "" # No default (required) restapi_version: v65.0 request_timeout: 30s max_retries: 10 query_type: rest query: "" cdc_enabled: false cdc_objects: [] cdc_batch_size: 100 cdc_buffer_size: 1000 cdc_replay_preset: latest pubsub_topic: "" grpc_reconnect_base_delay: 500ms grpc_reconnect_max_delay: 30s grpc_reconnect_max_attempts: 0 grpc_shutdown_timeout: 10s cache_resource: salesforce_checkpoint parallel_fetch: 1 query_batch_size: 2000 rest_objects: [] ``` ## [](#fields)Fields ### [](#cache_resource)`cache_resource` Name of the cache resource used for checkpointing state (must be defined in `cache_resources`). **Type**: `string` **Default**: `salesforce_checkpoint` ### [](#cdc_batch_size)`cdc_batch_size` Number of CDC events to request per gRPC fetch. **Type**: `int` **Default**: `100` ### [](#cdc_buffer_size)`cdc_buffer_size` Size of the internal CDC event buffer. **Type**: `int` **Default**: `1000` ### [](#cdc_enabled)`cdc_enabled` Enables Change Data Capture streaming after the REST snapshot completes. **Type**: `bool` **Default**: `false` ### [](#cdc_objects)`cdc_objects[]` sObject types to capture changes for (for example, `["Account", "Contact"]`). When empty, subscribes to `/data/ChangeEvents` for all objects. **Type**: `array` **Default**: `[]` ### [](#cdc_replay_preset)`cdc_replay_preset` CDC replay preset when no checkpoint exists: `latest` (default) or `earliest`. **Type**: `string` **Default**: `latest` ### [](#client_id)`client_id` Client ID for the Salesforce Connected App. **Type**: `string` ### [](#client_secret)`client_secret` Client secret for the Salesforce Connected App. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#grpc_reconnect_base_delay)`grpc_reconnect_base_delay` Base delay for gRPC reconnection backoff. **Type**: `string` **Default**: `500ms` ### [](#grpc_reconnect_max_attempts)`grpc_reconnect_max_attempts` Maximum number of gRPC reconnection attempts (0 = unlimited). **Type**: `int` **Default**: `0` ### [](#grpc_reconnect_max_delay)`grpc_reconnect_max_delay` Maximum delay for gRPC reconnection backoff. **Type**: `string` **Default**: `30s` ### [](#grpc_shutdown_timeout)`grpc_shutdown_timeout` Timeout for graceful gRPC client shutdown. **Type**: `string` **Default**: `10s` ### [](#max_retries)`max_retries` Maximum number of retries on a 429 Too Many Requests error. **Type**: `int` **Default**: `10` ### [](#org_url)`org_url` Salesforce instance base URL (for example, [https://your-domain.salesforce.com](https://your-domain.salesforce.com)). **Type**: `string` ### [](#parallel_fetch)`parallel_fetch` Number of sObjects to fetch concurrently during the REST snapshot (no-query mode). Higher values improve throughput but consume more API quota. **Type**: `int` **Default**: `1` ### [](#pubsub_topic)`pubsub_topic` Arbitrary Pub/Sub API topic (for example, `/event/MyEvent__e`). When set, overrides `cdc_objects` for topic selection. **Type**: `string` **Default**: `""` ### [](#query)`query` Optional SOQL (REST) or GraphQL query. When empty, all sObjects are fetched using REST. **Type**: `string` **Default**: `""` ### [](#query_batch_size)`query_batch_size` Number of records Salesforce returns per query page (200-2000). Lower values reduce individual response size and avoid timeouts on wide sObjects. **Type**: `int` **Default**: `2000` ### [](#query_type)`query_type` API mode: `rest` (default) or `graphql`. **Type**: `string` **Default**: `rest` ### [](#request_timeout)`request_timeout` Timeout for HTTP requests. **Type**: `string` **Default**: `30s` ### [](#rest_objects)`rest_objects[]` Limits the REST snapshot to only these sObject types (for example, `["Account", "Contact"]`). When empty, all queryable sObjects are fetched. **Type**: `array` **Default**: `[]` ### [](#restapi_version)`restapi_version` Salesforce REST API version to use (for example, v65.0). **Type**: `string` **Default**: `v65.0` --- # Page 283: schema_registry_decode **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/schema_registry_decode.md --- # schema\_registry\_decode --- title: schema_registry_decode latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/schema_registry_decode page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/schema_registry_decode.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/schema_registry_decode.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/schema_registry_decode/ "View the Cloud version of this component"), Self-Managed Automatically decodes and validates messages with schemas from a Confluent Schema Registry service. This processor uses the [Franz Kafka Schema Registry client](https://github.com/twmb/franz-go/tree/master/pkg/sr). #### Common ```yml processors: label: "" schema_registry_decode: avro: raw_unions: "" # No default (optional) preserve_logical_types: false translate_kafka_connect_types: false mapping: "" # No default (optional) store_schema_metadata: "" # No default (optional) protobuf: use_proto_names: false use_enum_numbers: false emit_unpopulated: false emit_default_values: false serialize_to_json: true cache_duration: 10m url: "" # No default (required) default_schema_id: "" # No default (optional) ``` #### Advanced ```yml processors: label: "" schema_registry_decode: avro: raw_unions: "" # No default (optional) preserve_logical_types: false translate_kafka_connect_types: false mapping: "" # No default (optional) store_schema_metadata: "" # No default (optional) protobuf: use_proto_names: false use_enum_numbers: false emit_unpopulated: false emit_default_values: false serialize_to_json: true cache_duration: 10m url: "" # No default (required) default_schema_id: "" # No default (optional) oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} tls: skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] ``` Decodes messages automatically from a schema stored within a [Confluent Schema Registry service](https://docs.confluent.io/platform/current/schema-registry/index.html) by extracting a schema ID from the message and obtaining the associated schema from the registry. If a message fails to match against the schema then it will remain unchanged and the error can be caught using [error-handling methods](../../../configuration/error_handling/). Avro, Protobuf and JSON schemas are supported, all are capable of expanding from schema references as of v4.22.0. ## [](#avro-json-format)Avro JSON format By default, this processor expects documents formatted as [Avro JSON](https://avro.apache.org/docs/current/specification/) when decoding with Avro schemas. In this format, the value of a union is encoded in JSON as follows: - If the union’s type is `null`, it is encoded as a JSON `null`. - Otherwise, the union is encoded as a JSON object with one name/value pair. The name is the type’s name, and the value is the recursively-encoded value. The user-specified name is used for Avro’s named types (record, fixed, or enum). For other types, the type name is used. For example, the union schema `["null","string","Transaction"]`, where `Transaction` is a record name, would encode: - `null` as a JSON `null` - The string `"a"` as `{"string": "a"}` - A `Transaction` instance as `{"Transaction": {…​}}`, where `{…​}` indicates the JSON encoding of a `Transaction` instance Alternatively, you can create documents in [standard/raw JSON format](https://pkg.go.dev/github.com/linkedin/goavro/v2#NewCodecForStandardJSONFull) by setting the field [`avro.raw_unions`](#avro-raw_unions) to `true`. ## [](#protobuf-format)Protobuf format This processor decodes Protobuf messages to JSON documents. For more information about the JSON mapping of Protobuf messages, see the [Protocol Buffers documentation](https://developers.google.com/protocol-buffers/docs/proto3#json). ## [](#metadata)Metadata This processor adds the following metadata to processed messages: - `schema_id`: The ID of the schema in the schema registry associated with the message. ## [](#fields)Fields ### [](#avro)`avro` Configuration for how to decode schemas that are of type AVRO. **Type**: `object` ### [](#avro-mapping)`avro.mapping` Define a custom mapping to apply to the JSON representation of Avro schemas. You can use mappings to convert custom types emitted by other tools, such as Debezium, into standard Avro types. **Type**: `string` ```yaml # Examples: mapping: |- map isDebeziumTimestampType { root = this.type == "long" && this."connect.name" == "io.debezium.time.Timestamp" && !this.exists("logicalType") } map debeziumTimestampToAvroTimestamp { let mapped_fields = this.fields.or([]).map_each(item -> item.apply("debeziumTimestampToAvroTimestamp")) root = match { this.type == "record" => this.assign({"fields": $mapped_fields}) this.type.type() == "array" => this.assign({"type": this.type.map_each(item -> item.apply("debeziumTimestampToAvroTimestamp"))}) # Add a logical type so that it's decoded as a timestamp instead of a long. this.type.type() == "object" && this.type.apply("isDebeziumTimestampType") => this.merge({"type":{"logicalType": "timestamp-millis"}}) _ => this } } root = this.apply("debeziumTimestampToAvroTimestamp") ``` ### [](#avro-preserve_logical_types)`avro.preserve_logical_types` Choose whether to: - Transform logical types into their primitive type (default). For example, decimals become raw bytes and timestamps become plain integers. - Preserve logical types. Set to `true` to preserve logical types. **Type**: `bool` **Default**: `false` ### [](#avro-raw_unions)`avro.raw_unions` Whether Avro messages should be decoded into normal JSON (JSON that meets the expectations of regular internet JSON) rather than [Avro JSON](https://avro.apache.org/docs/current/specification/). If set to `false`, Avro messages are decoded as [Avro JSON](https://pkg.go.dev/github.com/linkedin/goavro/v2#NewCodec). For example, the union schema `["null","string","Transaction"]`, where `Transaction` is a record name, would be decoded as: - A `null` as a JSON `null` - The string `"a"` as `{"string": "a"}` - A `Transaction` instance as `{"Transaction": {…​}}`, where `{…​}` indicates the JSON encoding of a `Transaction` instance. If set to `true`, Avro messages are decoded as [standard JSON](https://pkg.go.dev/github.com/linkedin/goavro/v2#NewCodecForStandardJSONFull). For example, the same union schema `["null","string","Transaction"]` is decoded as: - A `null` as JSON `null` - The string `"a"` as `"a"` - A `Transaction` instance as `{…​}`, where `{…​}` indicates the JSON encoding of a `Transaction` instance. For more details on the difference between standard JSON and Avro JSON, see the [comment in Goavro](https://github.com/linkedin/goavro/blob/5ec5a5ee7ec82e16e6e2b438d610e1cab2588393/union.go#L224-L249) and the [underlying library used for Avro serialization](https://github.com/linkedin/goavro). **Type**: `bool` ### [](#avro-store_schema_metadata)`avro.store_schema_metadata` Optionally store the schema used to decode messages as a metadata field under the given name. This field can later be referenced in other components such as a `parquet_encode` processor in order to automatically infer their schema. **Type**: `string` ### [](#avro-translate_kafka_connect_types)`avro.translate_kafka_connect_types` Only valid if preserve\_logical\_types is true. This decodes various Kafka Connect types into their bloblang equivalents when not representable by standard logical types according to the Avro standard. Types that are currently translated: | Type Name | Bloblang Type | Description | | --- | --- | --- | | io.debezium.time.Date | timestamp | Date without time (days since epoch) | | io.debezium.time.Timestamp | timestamp | Timestamp without timezone (milliseconds since epoch) | | io.debezium.time.MicroTimestamp | timestamp | Timestamp with microsecond precision | | io.debezium.time.NanoTimestamp | timestamp | Timestamp with nanosecond precision | | io.debezium.time.ZonedTimestamp | timestamp | Timestamp with timezone (ISO-8601 format) | | io.debezium.time.Year | timestamp at January 1st at 00:00:00 | Year value | | io.debezium.time.Time | timestamp at the unix epoch | Time without date (milliseconds past midnight) | | io.debezium.time.MicroTime | timestamp at the unix epoch | Time with microsecond precision | | io.debezium.time.NanoTime | timestamp at the unix epoch | Time with nanosecond precision | **Type**: `bool` **Default**: `false` ### [](#basic_auth)`basic_auth` Allows you to specify basic authentication. Requires version 4.7.0 or later. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#cache_duration)`cache_duration` The duration after which a cached schema is considered stale and is removed from the cache. **Type**: `string` **Default**: `10m` ```yaml # Examples: cache_duration: 1h # --- cache_duration: 5m ``` ### [](#default_schema_id)`default_schema_id` This schema ID is used when a message’s schema header cannot be read (`ErrBadHeader`). If this value is not set, schema header errors are returned. This configuration does not work with protobuf schemas. > 💡 **TIP** > > You can also use the [`with_schema_registry_header`](../../../guides/bloblang/functions/#with_schema_registry_header) bloblang function to add a schema ID to messages. **Type**: `int` ### [](#jwt)`jwt` Beta Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWT tokens provide secure, stateless authentication between services. Requires version 4.7.0 or later. **Type**: `object` ### [](#jwt-claims)`jwt.claims` A value used to identify the claims that issued the JWT. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` The cryptographic algorithm used to sign the JWT token. Supported algorithms include RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#oauth)`oauth` Configure OAuth version 1.0 authentication for secure API access. Requires version 4.7.0 or later. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` A value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` A secret provided in order to establish ownership of a given access token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` A value used to identify the client to the service provider. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` A secret used to establish ownership of the consumer key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#protobuf)`protobuf` Configuration for how to decode schemas that are of type PROTOBUF. **Type**: `object` ### [](#protobuf-emit_default_values)`protobuf.emit_default_values` Whether to emit default-valued primitive fields, empty lists, and empty maps. emit\_unpopulated takes precedence over emit\_default\_values **Type**: `bool` **Default**: `false` ### [](#protobuf-emit_unpopulated)`protobuf.emit_unpopulated` Whether to emit unpopulated fields. It does not emit unpopulated oneof fields or unpopulated extension fields. **Type**: `bool` **Default**: `false` ### [](#protobuf-serialize_to_json)`protobuf.serialize_to_json` If messages should be serialized to JSON bytes. If false then the message is kept in decoded form, which means that 64 bit integers are not converted to strings and types for bytes and google.protobuf.Timestamp are preserved (as they are not serialized to JSON strings). **Type**: `bool` **Default**: `true` ### [](#protobuf-use_enum_numbers)`protobuf.use_enum_numbers` Emits enum values as numbers. **Type**: `bool` **Default**: `false` ### [](#protobuf-use_proto_names)`protobuf.use_proto_names` Use proto field name instead of lowerCamelCase name. **Type**: `bool` **Default**: `false` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The base URL of the schema registry service. **Type**: `string` --- # Page 284: schema_registry_encode **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/schema_registry_encode.md --- # schema\_registry\_encode --- title: schema_registry_encode latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/schema_registry_encode page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/schema_registry_encode.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/schema_registry_encode.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/schema_registry_encode/ "View the Cloud version of this component"), Self-Managed Automatically encodes and validates messages with schemas from a Confluent Schema Registry service. This processor uses the [Franz Kafka Schema Registry client](https://github.com/twmb/franz-go/tree/master/pkg/sr). Introduced in version 3.58.0. #### Common ```yml processors: label: "" schema_registry_encode: url: "" # No default (required) subject: "" # No default (required) refresh_period: 10m schema_metadata: "" format: "" # No default (optional) avro: raw_json: "" # No default (optional) record_name: "" namespace: "" ``` #### Advanced ```yml processors: label: "" schema_registry_encode: url: "" # No default (required) subject: "" # No default (required) refresh_period: 10m schema_metadata: "" format: "" # No default (optional) normalize: true avro: raw_json: "" # No default (optional) record_name: "" namespace: "" oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} tls: skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] ``` Encodes messages automatically from schemas obtains from a [Confluent Schema Registry service](https://docs.confluent.io/platform/current/schema-registry/index.html) by polling the service for the latest schema version for target subjects. If a message fails to encode under the schema then it will remain unchanged and the error can be caught using [error-handling methods](../../../configuration/error_handling/). Avro, Protobuf and JSON schemas are supported, all are capable of expanding from schema references as of v4.22.0. ## [](#avro-json-format)Avro JSON format By default, this processor expects documents formatted as [Avro JSON](https://avro.apache.org/docs/current/specification/) when encoding with Avro schemas. In this format, the value of a union is encoded in JSON as follows: - If the union’s type is `null`, it is encoded as a JSON `null`. - Otherwise, the union is encoded as a JSON object with one name/value pair. The name is the type’s name, and the value is the recursively-encoded value. The user-specified name is used for Avro’s named types (record, fixed, or enum). For other types, the type name is used. For example, the union schema `["null","string","Transaction"]`, where `Transaction` is a record name, would encode: - A `null` as a JSON `null` - The string `"a"` as `{"string": "a"}` - A `Transaction` instance as `{"Transaction": {…​}}`, where `{…​}` indicates the JSON encoding of a `Transaction` instance Alternatively, you can consume documents in [standard/raw JSON format](https://pkg.go.dev/github.com/linkedin/goavro/v2#NewCodecForStandardJSONFull) by setting the field [`avro_raw_json`](#avro_raw_json) to `true`. ### [](#known-issues)Known issues Important! There is an outstanding issue in the [avro serializing library](https://github.com/linkedin/goavro) that Redpanda Connect uses which means it [doesn’t encode logical types correctly](https://github.com/linkedin/goavro/issues/252). It’s still possible to encode logical types that are in-line with the spec if `avro_raw_json` is set to true, though now of course non-logical types will not be in-line with the spec. ## [](#protobuf-format)Protobuf format This processor encodes Protobuf messages either from any format parsed within Redpanda Connect (encoded as JSON by default), or from raw JSON documents. For more information about the JSON mapping of Protobuf messages, see the [Protocol Buffers documentation](https://developers.google.com/protocol-buffers/docs/proto3#json). ### [](#multiple-message-support)Multiple message support When a target subject presents a Protobuf schema that contains multiple messages it becomes ambiguous which message definition a given input data should be encoded against. In such scenarios Redpanda Connect will attempt to encode the data against each of them and select the first to successfully match against the data, this process currently **ignores all nested message definitions**. In order to speed up this exhaustive search the last known successful message will be attempted first for each subsequent input. We will be considering alternative approaches in future so please [get in touch](https://redpanda.com/slack) with thoughts and feedback. ## [](#fields)Fields ### [](#avro)`avro` Configuration for Avro encoding. **Type**: `object` ### [](#avro-namespace)`avro.namespace` The Avro namespace for the root record type when encoding from a common schema (schema\_metadata mode). **Type**: `string` **Default**: `""` ### [](#avro-raw_json)`avro.raw_json` Whether messages encoded in Avro format should be parsed as normal JSON rather than Avro JSON. Overrides the deprecated top-level `avro_raw_json` when set. **Type**: `bool` ### [](#avro-record_name)`avro.record_name` The name to use for the root Avro record type when encoding from a common schema (schema\_metadata mode). If empty, derived from the subject. **Type**: `string` **Default**: `""` ### [](#basic_auth)`basic_auth` Allows you to specify basic authentication. Requires version 4.7.0 or later. **Type**: `object` ### [](#basic_auth-enabled)`basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#basic_auth-password)`basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#basic_auth-username)`basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#format)`format` The encoding format to use when converting a common schema from metadata. Required when `schema_metadata` is set. **Type**: `string` **Options**: `avro`, `json_schema` ### [](#jwt)`jwt` Beta Configure JSON Web Token (JWT) authentication. This feature is in beta and may change in future releases. JWT tokens provide secure, stateless authentication between services. Requires version 4.7.0 or later. **Type**: `object` ### [](#jwt-claims)`jwt.claims` A value used to identify the claims that issued the JWT. **Type**: `object` **Default**: `{}` ### [](#jwt-enabled)`jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#jwt-headers)`jwt.headers` Additional key-value pairs to include in the JWT header (optional). These headers provide extra metadata for JWT processing. **Type**: `object` **Default**: `{}` ### [](#jwt-private_key_file)`jwt.private_key_file` Path to a file containing the PEM-encoded private key using PKCS#1 or PKCS#8 format. The private key must be compatible with the algorithm specified in the `signing_method` field. **Type**: `string` **Default**: `""` ### [](#jwt-signing_method)`jwt.signing_method` The cryptographic algorithm used to sign the JWT token. Supported algorithms include RS256, RS384, RS512, and EdDSA. This algorithm must be compatible with the private key specified in the `private_key_file` field. **Type**: `string` **Default**: `""` ### [](#normalize)`normalize` Whether to normalize the schema before registering with the schema registry (schema\_metadata mode only). **Type**: `bool` **Default**: `true` ### [](#oauth)`oauth` Configure OAuth version 1.0 authentication for secure API access. Requires version 4.7.0 or later. **Type**: `object` ### [](#oauth-access_token)`oauth.access_token` A value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#oauth-access_token_secret)`oauth.access_token_secret` A secret provided in order to establish ownership of a given access token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-consumer_key)`oauth.consumer_key` A value used to identify the client to the service provider. **Type**: `string` **Default**: `""` ### [](#oauth-consumer_secret)`oauth.consumer_secret` A secret used to establish ownership of the consumer key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#oauth-enabled)`oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#refresh_period)`refresh_period` The period after which a schema is refreshed for each subject, this is done by polling the schema registry service. **Type**: `string` **Default**: `10m` ```yaml # Examples: refresh_period: 60s # --- refresh_period: 1h ``` ### [](#schema_metadata)`schema_metadata` When set, the processor reads a schema in benthos common schema format from this metadata key on each message, converts it to the format specified by `format`, registers it with the schema registry under the configured subject, and encodes the message. When empty (the default), the processor pulls the latest schema from the registry instead. **Type**: `string` **Default**: `""` ### [](#subject)`subject` The schema subject to derive schemas from. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: subject: foo # --- subject: ${! meta("kafka_topic") } ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#url)`url` The base URL of the schema registry service. **Type**: `string` --- # Page 285: select_parts **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/select_parts.md --- # select\_parts --- title: select_parts latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/select_parts page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/select_parts.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/select_parts.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/select_parts/ "View the Cloud version of this component"), Self-Managed Cherry pick a set of messages from a batch by their index. Indexes larger than the number of messages are simply ignored. ```yml # Config fields, showing default values label: "" select_parts: parts: [] ``` The selected parts are added to the new message batch in the same order as the selection array. E.g. with 'parts' set to \[ 2, 0, 1 \] and the message parts \[ '0', '1', '2', '3' \], the output will be \[ '2', '0', '1' \]. If none of the selected parts exist in the input batch (resulting in an empty output message) the batch is dropped entirely. Message indexes can be negative, and if so the part will be selected from the end counting backwards starting from -1. E.g. if index = -1 then the selected part will be the last part of the message, if index = -2 then the part before the last element with be selected, and so on. This processor is only applicable to [batched messages](../../../configuration/batching/). ## [](#fields)Fields ### [](#parts)`parts[]` An array of message indexes of a batch. Indexes can be negative, and if so the part will be selected from the end counting backwards starting from -1. **Type**: `int` **Default**: `[]` --- # Page 286: sentry_capture **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/sentry_capture.md --- # sentry\_capture --- title: sentry_capture latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/sentry_capture page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/sentry_capture.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/sentry_capture.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Available in:** Self-Managed Captures log events from messages and submits them to [Sentry](https://sentry.io/). Introduced in version 4.16.0. ```yml # Config fields, showing default values label: "" sentry_capture: dsn: "" message: webhook event received # No default (required) context: 'root = {"order": {"product_id": "P93174", "quantity": 5}}' # No default (optional) tags: {} # No default (optional) environment: "" release: "" level: INFO transport_mode: async flush_timeout: 5s sampling_rate: 1 ``` ## [](#fields)Fields ### [](#context)`context` A mapping that must evaluate to an object-of-objects or `deleted()`. If this mapping produces a value, then it is set on a sentry event as additional context. **Type**: `string` ```yaml # Examples: context: root = {"order": {"product_id": "P93174", "quantity": 5}} # --- context: root = deleted() ``` ### [](#dsn)`dsn` The DSN address to send sentry events to. If left empty, then SENTRY\_DSN is used. **Type**: `string` **Default**: `""` ### [](#environment)`environment` The environment to be sent with events. If left empty, then SENTRY\_ENVIRONMENT is used. **Type**: `string` **Default**: `""` ### [](#extras)`extras` A mapping that must evaluate to an object. If this mapping produces a value, then it is set on a sentry event as extras. **Type**: `string` ```yaml # Examples: extras: root.foo = "bar" # --- extras: root = this.without("password") ``` ### [](#flush_timeout)`flush_timeout` The duration to wait when closing the processor to flush any remaining enqueued events. **Type**: `string` **Default**: `5s` ### [](#level)`level` Sets the level on sentry events similar to logging levels. **Type**: `string` **Default**: `INFO` **Options**: `DEBUG`, `INFO`, `WARN`, `ERROR`, `FATAL` ### [](#message)`message` A message to set on the sentry event This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ```yaml # Examples: message: webhook event received # --- message: failed to find product in database: ${! error() } ``` ### [](#release)`release` The version of the code deployed to an environment. If left empty, then the Sentry client will attempt to detect the release from the environment. **Type**: `string` **Default**: `""` ### [](#sampling_rate)`sampling_rate` The rate at which events are sent to the server. A value of 0 disables capturing sentry events entirely. A value of 1 results in sending all events to Sentry. Any value in between results sending some percentage of events. **Type**: `float` **Default**: `1` ### [](#tags)`tags` Sets key/value string tags on an event. Unlike context, these are indexed and searchable on Sentry but have length limitations. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#transport_mode)`transport_mode` Determines how events are sent. A sync transport will block when sending each event until a response is received from the Sentry server. The recommended async transport will enqueue events in a buffer and send them in the background. **Type**: `string` **Default**: `async` **Options**: `async`, `sync` --- # Page 287: slack_thread **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/slack_thread.md --- # slack\_thread --- title: slack_thread latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/slack_thread page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/slack_thread.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/slack_thread.adoc page-git-created-date: "2025-05-02" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/slack_thread/ "View the Cloud version of this component"), Self-Managed **License**: This component requires an [enterprise license](https://docs.redpanda.com/redpanda-connect/get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that's valid for 30 days. Reads a Slack thread using the Slack API method [conversations.replies](https://api.slack.com/methods/conversations.replies). Introduced in version 4.52.0. ```yml # Common configuration fields, showing default values label: "" slack_thread: bot_token: "" # No default (required) channel_id: "" # No default (required) thread_ts: "" # No default (required) ``` ## [](#fields)Fields ### [](#bot_token)`bot_token` Your Slack bot user’s OAuth token, which must have the correct permissions to read messages from the Slack channel specified in `channel_id`. **Type**: `string` ### [](#channel_id)`channel_id` The encoded ID of the Slack channel from which to read threads. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` ### [](#thread_ts)`thread_ts` The timestamp of the parent message of the thread you want to read. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 288: sleep **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/sleep.md --- # sleep --- title: sleep latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/sleep page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/sleep.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/sleep.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/sleep/ "View the Cloud version of this component"), Self-Managed Sleep for a period of time specified as a duration string for each message. This processor will interpolate functions within the `duration` field, you can find a list of functions [here](../../../configuration/interpolation/#bloblang-queries). ```yml # Config fields, showing default values label: "" sleep: duration: "" # No default (required) ``` ## [](#fields)Fields ### [](#duration)`duration` The duration of time to sleep for each execution. This field supports [interpolation functions](../../../configuration/interpolation/#bloblang-queries). **Type**: `string` --- # Page 289: split **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/split.md --- # split --- title: split latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/split page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/split.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/split.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/split/ "View the Cloud version of this component"), Self-Managed Breaks message batches (synonymous with multiple part messages) into smaller batches. The size of the resulting batches are determined either by a discrete size or, if the field `byte_size` is non-zero, then by total size in bytes (which ever limit is reached first). ```yml # Config fields, showing default values label: "" split: size: 1 byte_size: 0 ``` This processor is for breaking batches down into smaller ones. In order to break a single message out into multiple messages use the [`unarchive` processor](../unarchive/). If there is a remainder of messages after splitting a batch the remainder is also sent as a single batch. For example, if your target size was 10, and the processor received a batch of 95 message parts, the result would be 9 batches of 10 messages followed by a batch of 5 messages. ## [](#fields)Fields ### [](#byte_size)`byte_size` An optional target of total message bytes. **Type**: `int` **Default**: `0` ### [](#size)`size` The target number of messages. **Type**: `int` **Default**: `1` --- # Page 290: sql_insert **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/sql_insert.md --- # sql\_insert --- title: sql_insert latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/sql_insert page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/sql_insert.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/sql_insert.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-03-18" --- **Type:** Processor ▼ [Processor](/redpanda-connect/components/processors/sql_insert/)[Output](/redpanda-connect/components/outputs/sql_insert/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/sql_insert/ "View the Cloud version of this component"), Self-Managed Inserts rows into an SQL database for each message, and leaves the message unchanged. Introduced in version 3.59.0. #### Common ```yml processors: label: "" sql_insert: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) columns: [] # No default (required) args_mapping: "" # No default (required) ``` #### Advanced ```yml processors: label: "" sql_insert: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) columns: [] # No default (required) args_mapping: "" # No default (required) prefix: "" # No default (optional) suffix: "" # No default (optional) options: [] # No default (optional) init_files: [] # No default (optional) init_statement: "" # No default (optional) conn_max_idle_time: "" # No default (optional) conn_max_life_time: "" # No default (optional) conn_max_idle: 2 conn_max_open: "" # No default (optional) ``` If the insert fails to execute then the message will still remain unchanged and the error can be caught using [error handling methods](../../../configuration/error_handling/). ## [](#examples)Examples ### [](#table-insert-mysql)Table Insert (MySQL) Here we insert rows into a database by populating the columns id, name and topic with values extracted from messages and metadata: ```yaml pipeline: processors: - sql_insert: driver: mysql dsn: foouser:foopassword@tcp(localhost:3306)/foodb table: footable columns: [ id, name, topic ] args_mapping: | root = [ this.user.id, this.user.name, meta("kafka_topic"), ] ``` ## [](#dynamic-sql-operations)Dynamic SQL operations The `table` and `columns` fields are static strings that do not support Bloblang interpolation. For dynamic table names, dynamic column lists, DELETE operations, or any other SQL that `sql_insert` cannot express, use the [`sql_raw` processor](../sql_raw/) instead. To use Bloblang interpolation inside ``sql_raw’s `query`` field, you must enable `unsafe_dynamic_query: true`. > ⚠️ **CAUTION** > > Interpolating unsanitized values into a query can introduce SQL injection risks. Always validate or sanitize the interpolated value beforehand. ## [](#fields)Fields ### [](#args_mapping)`args_mapping` A [Bloblang mapping](../../../guides/bloblang/about/) which should evaluate to an array of values matching in size to the number of columns specified. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#columns)`columns[]` A list of columns to insert. **Type**: `array` ```yaml # Examples: columns: - foo - bar - baz ``` ### [](#conn_max_idle)`conn_max_idle` An optional maximum number of connections in the idle connection pool. If conn\_max\_open is greater than 0 but less than the new conn\_max\_idle, then the new conn\_max\_idle will be reduced to match the conn\_max\_open limit. If `value ⇐ 0`, no idle connections are retained. The default max idle connections is currently 2. This may change in a future release. **Type**: `int` **Default**: `2` ### [](#conn_max_idle_time)`conn_max_idle_time` An optional maximum amount of time a connection may be idle. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections idle time. **Type**: `string` ### [](#conn_max_life_time)`conn_max_life_time` An optional maximum amount of time a connection may be reused. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections age. **Type**: `string` ### [](#conn_max_open)`conn_max_open` An optional maximum number of open connections to the database. If conn\_max\_idle is greater than 0 and the new conn\_max\_open is less than conn\_max\_idle, then conn\_max\_idle will be reduced to match the new conn\_max\_open limit. If `value ⇐ 0`, then there is no limit on the number of open connections. The default is 0 (unlimited). **Type**: `int` ### [](#driver)`driver` A database [driver](#drivers) to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks` ### [](#dsn)`dsn` A Data Source Name to identify the target database. #### [](#drivers)Drivers The following is a list of supported drivers, their placeholder style, and their respective DSN formats: | Driver | Data Source Name Format | | --- | --- | | clickhouse | clickhouse://[username[:password]@][netloc][:port]/dbname[?param1=value1&…​¶mN=valueN] | | mysql | [username[:password]@][protocol[(address)]]/dbname[?param1=value1&…​¶mN=valueN] | | postgres and pgx | postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&…​] | | mssql | sqlserver://[user[:password]@][netloc][:port][?database=dbname¶m1=value1&…​] | | sqlite | file:/path/to/filename.db[?param&=value1&…​] | | oracle | oracle://[username[:password]@][netloc][:port]/service_name?server=server2&server=server3 | | snowflake | username[:password]@account_identifier/dbname/schemaname[?param1=value&…​¶mN=valueN] | | trino | http[s]://user[:pass]@host[:port][?parameters] | | gocosmos | AccountEndpoint=;AccountKey=[;TimeoutMs=][;Version=][;DefaultDb/Db=][;AutoId=][;InsecureSkipVerify=] | | spanner | projects/[PROJECT]/instances/[INSTANCE]/databases/[DATABASE] | | databricks | token:@:/ | Please note that the `postgres` and `pgx` drivers enforce SSL by default, you can override this with the parameter `sslmode=disable` if required. The `pgx` driver is an alternative to the standard `postgres` (pq) driver and comes with extra functionality such as support for array insertion. The `snowflake` driver supports multiple DSN formats. Please consult [the docs](https://pkg.go.dev/github.com/snowflakedb/gosnowflake#hdr-Connection_String) for more details. For [key pair authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication), the DSN has the following format: `@//?warehouse=&role=&authenticator=snowflake_jwt&privateKey=`, where the value for the `privateKey` parameter can be constructed from an unencrypted RSA private key file `rsa_key.p8` using `openssl enc -d -base64 -in rsa_key.p8 | basenc --base64url -w0` (you can use `gbasenc` instead of `basenc` on OSX if you install `coreutils` via Homebrew). If you have a password-encrypted private key, you can decrypt it using `openssl pkcs8 -in rsa_key_encrypted.p8 -out rsa_key.p8`. Also, make sure fields such as the username are URL-encoded. The [`gocosmos`](https://pkg.go.dev/github.com/microsoft/gocosmos) driver is still experimental, but it has support for [hierarchical partition keys](https://learn.microsoft.com/en-us/azure/cosmos-db/hierarchical-partition-keys) as well as [cross-partition queries](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-query-container#cross-partition-query). Please refer to the [SQL notes](https://github.com/microsoft/gocosmos/blob/main/SQL.md) for details. **Type**: `string` ```yaml # Examples: dsn: clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60 # --- dsn: foouser:foopassword@tcp(localhost:3306)/foodb # --- dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable # --- dsn: oracle://foouser:foopass@localhost:1521/service_name # --- dsn: token:dapi1234567890ab@dbc-a1b2345c-d6e7.cloud.databricks.com:443/sql/1.0/warehouses/abc123def456 ``` ### [](#init_files)`init_files[]` An optional list of file paths containing SQL statements to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Glob patterns are supported, including super globs (double star). Care should be taken to ensure that the statements are idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If a statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `array` ```yaml # Examples: init_files: - ./init/*.sql # --- init_files: - ./foo.sql - ./bar.sql ``` ### [](#init_statement)`init_statement` An optional SQL statement to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Care should be taken to ensure that the statement is idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If the statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `string` ```yaml # Examples: init_statement: |- CREATE TABLE IF NOT EXISTS some_table ( foo varchar(50) not null, bar integer, baz varchar(50), primary key (foo) ) WITHOUT ROWID; ``` ### [](#options)`options[]` A list of keyword options to add before the INTO clause of the query. **Type**: `array` ```yaml # Examples: options: - DELAYED - IGNORE ``` ### [](#prefix)`prefix` An optional prefix to prepend to the insert query (before INSERT). **Type**: `string` ### [](#suffix)`suffix` An optional suffix to append to the insert query. **Type**: `string` ```yaml # Examples: suffix: ON CONFLICT (name) DO NOTHING ``` ### [](#table)`table` The table to insert to. **Type**: `string` ```yaml # Examples: table: foo ``` --- # Page 291: sql_raw **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/sql_raw.md --- # sql\_raw --- title: sql_raw latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/sql_raw page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/sql_raw.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/sql_raw.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Processor ▼ [Processor](/redpanda-connect/components/processors/sql_raw/)[Input](/redpanda-connect/components/inputs/sql_raw/)[Output](/redpanda-connect/components/outputs/sql_raw/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/sql_raw/ "View the Cloud version of this component"), Self-Managed Runs an arbitrary SQL query against a database and (optionally) returns the result as an array of objects, one for each row returned. Introduced in version 3.65.0. #### Common ```yml processors: label: "" sql_raw: driver: "" # No default (required) dsn: "" # No default (required) query: "" # No default (optional) args_mapping: "" # No default (optional) exec_only: "" # No default (optional) queries: [] # No default (optional) ``` #### Advanced ```yml processors: label: "" sql_raw: driver: "" # No default (required) dsn: "" # No default (required) query: "" # No default (optional) unsafe_dynamic_query: false args_mapping: "" # No default (optional) exec_only: "" # No default (optional) queries: [] # No default (optional) init_files: [] # No default (optional) init_statement: "" # No default (optional) conn_max_idle_time: "" # No default (optional) conn_max_life_time: "" # No default (optional) conn_max_idle: 2 conn_max_open: "" # No default (optional) ``` If the query fails to execute then the message will remain unchanged and the error can be caught using [error handling methods](../../../configuration/error_handling/). For some scenarios where you might use this processor, see [Examples](#examples). ## [](#fields)Fields ### [](#args_mapping)`args_mapping` An optional [Bloblang mapping](../../../guides/bloblang/about/) that includes the same number of values in an array as the placeholder arguments in the [`query`](#query) field. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#conn_max_idle)`conn_max_idle` An optional maximum number of connections in the idle connection pool. If conn\_max\_open is greater than 0 but less than the new conn\_max\_idle, then the new conn\_max\_idle will be reduced to match the conn\_max\_open limit. If `value ⇐ 0`, no idle connections are retained. The default max idle connections is currently 2. This may change in a future release. **Type**: `int` **Default**: `2` ### [](#conn_max_idle_time)`conn_max_idle_time` An optional maximum amount of time a connection may be idle. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections idle time. **Type**: `string` ### [](#conn_max_life_time)`conn_max_life_time` An optional maximum amount of time a connection may be reused. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections age. **Type**: `string` ### [](#conn_max_open)`conn_max_open` An optional maximum number of open connections to the database. If conn\_max\_idle is greater than 0 and the new conn\_max\_open is less than conn\_max\_idle, then conn\_max\_idle will be reduced to match the new conn\_max\_open limit. If `value ⇐ 0`, then there is no limit on the number of open connections. The default is 0 (unlimited). **Type**: `int` ### [](#driver)`driver` A database [driver](#drivers) to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks` ### [](#dsn)`dsn` A Data Source Name to identify the target database. #### [](#drivers)Drivers The following is a list of supported drivers, their placeholder style, and their respective DSN formats: | Driver | Data Source Name Format | | --- | --- | | clickhouse | clickhouse://[username[:password]@][netloc][:port]/dbname[?param1=value1&…​¶mN=valueN] | | mysql | [username[:password]@][protocol[(address)]]/dbname[?param1=value1&…​¶mN=valueN] | | postgres and pgx | postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&…​] | | mssql | sqlserver://[user[:password]@][netloc][:port][?database=dbname¶m1=value1&…​] | | sqlite | file:/path/to/filename.db[?param&=value1&…​] | | oracle | oracle://[username[:password]@][netloc][:port]/service_name?server=server2&server=server3 | | snowflake | username[:password]@account_identifier/dbname/schemaname[?param1=value&…​¶mN=valueN] | | trino | http[s]://user[:pass]@host[:port][?parameters] | | gocosmos | AccountEndpoint=;AccountKey=[;TimeoutMs=][;Version=][;DefaultDb/Db=][;AutoId=][;InsecureSkipVerify=] | | spanner | projects/[PROJECT]/instances/[INSTANCE]/databases/[DATABASE] | | databricks | token:@:/ | Please note that the `postgres` and `pgx` drivers enforce SSL by default, you can override this with the parameter `sslmode=disable` if required. The `pgx` driver is an alternative to the standard `postgres` (pq) driver and comes with extra functionality such as support for array insertion. The `snowflake` driver supports multiple DSN formats. Please consult [the docs](https://pkg.go.dev/github.com/snowflakedb/gosnowflake#hdr-Connection_String) for more details. For [key pair authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication), the DSN has the following format: `@//?warehouse=&role=&authenticator=snowflake_jwt&privateKey=`, where the value for the `privateKey` parameter can be constructed from an unencrypted RSA private key file `rsa_key.p8` using `openssl enc -d -base64 -in rsa_key.p8 | basenc --base64url -w0` (you can use `gbasenc` instead of `basenc` on OSX if you install `coreutils` via Homebrew). If you have a password-encrypted private key, you can decrypt it using `openssl pkcs8 -in rsa_key_encrypted.p8 -out rsa_key.p8`. Also, make sure fields such as the username are URL-encoded. The [`gocosmos`](https://pkg.go.dev/github.com/microsoft/gocosmos) driver is still experimental, but it has support for [hierarchical partition keys](https://learn.microsoft.com/en-us/azure/cosmos-db/hierarchical-partition-keys) as well as [cross-partition queries](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-query-container#cross-partition-query). Please refer to the [SQL notes](https://github.com/microsoft/gocosmos/blob/main/SQL.md) for details. **Type**: `string` ```yaml # Examples: dsn: clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60 # --- dsn: foouser:foopassword@tcp(localhost:3306)/foodb # --- dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable # --- dsn: oracle://foouser:foopass@localhost:1521/service_name # --- dsn: token:dapi1234567890ab@dbc-a1b2345c-d6e7.cloud.databricks.com:443/sql/1.0/warehouses/abc123def456 ``` ### [](#exec_only)`exec_only` Whether to discard the [`query`](#query) result. Set to `true` to leave the message contents unchanged, which is useful when you are executing inserts, updates, and so on. By default, the message contents are kept for the last query executed, and previous queries don’t change the results. **Type**: `bool` ### [](#init_files)`init_files[]` An optional list of file paths containing SQL statements to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Glob patterns are supported, including super globs (double star). Care should be taken to ensure that the statements are idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If a statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `array` ```yaml # Examples: init_files: - ./init/*.sql # --- init_files: - ./foo.sql - ./bar.sql ``` ### [](#init_statement)`init_statement` An optional SQL statement to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Care should be taken to ensure that the statement is idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If the statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `string` ```yaml # Examples: init_statement: |- CREATE TABLE IF NOT EXISTS some_table ( foo varchar(50) not null, bar integer, baz varchar(50), primary key (foo) ) WITHOUT ROWID; ``` ### [](#queries)`queries[]` A list of database statements to run in addition to your main [`query`](#query). If you specify multiple queries, they are executed within a single transaction. For more information, see [Examples](#examples). **Type**: `object` ### [](#queries-args_mapping)`queries[].args_mapping` An optional [Bloblang mapping](../../../guides/bloblang/about/) which should evaluate to an array of values matching in size to the number of placeholder arguments in the field `query`. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#queries-exec_only)`queries[].exec_only` Whether the query result should be discarded. When set to `true` the message contents will remain unchanged, which is useful in cases where you are executing inserts, updates, etc. By default this is true for the last query, and previous queries don’t change the results. If set to true for any query but the last one, the subsequent `args_mappings` input is overwritten. **Type**: `bool` ### [](#queries-query)`queries[].query` The query to execute. The style of placeholder to use depends on the driver, some drivers require question marks (`?`) whereas others expect incrementing dollar signs (`$1`, `$2`, and so on) or colons (`:1`, `:2` and so on). The style to use is outlined in this table: | Driver | Placeholder Style | |---|---| | `clickhouse` | Dollar sign | | `mysql` | Question mark | | `postgres` | Dollar sign | | `pgx` | Dollar sign | | `mssql` | Question mark | | `sqlite` | Question mark | | `oracle` | Colon | | `snowflake` | Question mark | | `trino` | Question mark | | `gocosmos` | Colon | **Type**: `string` ### [](#query)`query` The query to execute. You must include the correct placeholders for the specified database driver. Some drivers use question marks (`?`), whereas others expect incrementing dollar signs (`$1`, `$2`, and so on) or colons (`:1`, `:2`, and so on). | Driver | Placeholder Style | | --- | --- | | clickhouse | Dollar sign ($) | | gocosmos | Colon (:) | | mysql | Question mark (?) | | mssql | Question mark (?) | | oracle | Colon (:) | | postgres | Dollar sign ($) | | snowflake | Question mark (?) | | spanner | Question mark (?) | | sqlite | Question mark (?) | | trino | Question mark (?) | **Type**: `string` ```yaml # Examples: query: INSERT INTO footable (foo, bar, baz) VALUES (?, ?, ?); # --- query: SELECT * FROM footable WHERE user_id = $1; ``` ### [](#unsafe_dynamic_query)`unsafe_dynamic_query` Whether to enable [interpolation functions](../../../configuration/interpolation/#bloblang-queries) in the query. Great care should be made to ensure your queries are defended against injection attacks. **Type**: `bool` **Default**: `false` ## [](#examples)Examples ### [](#table-insert-mysql)Table Insert (MySQL) The following example inserts rows into the table footable with the columns foo, bar and baz populated with values extracted from messages. ```yaml pipeline: processors: - sql_raw: driver: mysql dsn: foouser:foopassword@tcp(localhost:3306)/foodb query: "INSERT INTO footable (foo, bar, baz) VALUES (?, ?, ?);" args_mapping: '[ document.foo, document.bar, meta("kafka_topic") ]' exec_only: true ``` ### [](#table-query-postgresql)Table Query (PostgreSQL) Here we query a database for columns of footable that share a `user_id` with the message field `user.id`. A [`branch` processor](../branch/) is used in order to insert the resulting array into the original message at the path `foo_rows`. ```yaml pipeline: processors: - branch: processors: - sql_raw: driver: postgres dsn: postgres://foouser:foopass@localhost:5432/testdb?sslmode=disable query: "SELECT * FROM footable WHERE user_id = $1;" args_mapping: '[ this.user.id ]' result_map: 'root.foo_rows = this' ``` ### [](#dynamically-creating-tables-postgresql)Dynamically Creating Tables (PostgreSQL) Here we query a database for columns of footable that share a `user_id` with the message field `user.id`. A [`branch` processor](../branch/) is used in order to insert the resulting array into the original message at the path `foo_rows`. ```yaml pipeline: processors: - mapping: | root = this # Prevent SQL injection when using unsafe_dynamic_query meta table_name = "\"" + metadata("table_name").replace_all("\"", "\"\"") + "\"" - sql_raw: driver: postgres dsn: postgres://localhost/postgres unsafe_dynamic_query: true queries: - query: | CREATE TABLE IF NOT EXISTS ${!metadata("table_name")} (id varchar primary key, document jsonb); - query: | INSERT INTO ${!metadata("table_name")} (id, document) VALUES ($1, $2) ON CONFLICT (id) DO UPDATE SET document = EXCLUDED.document; args_mapping: | root = [ this.id, this.document.string() ] ``` --- # Page 292: sql_select **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/sql_select.md --- # sql\_select --- title: sql_select latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/sql_select page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/sql_select.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/sql_select.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Type:** Processor ▼ [Processor](/redpanda-connect/components/processors/sql_select/)[Input](/redpanda-connect/components/inputs/sql_select/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/sql_select/ "View the Cloud version of this component"), Self-Managed Runs an SQL select query against a database and returns the result as an array of objects, one for each row returned, containing a key for each column queried and its value. Introduced in version 3.59.0. #### Common ```yml processors: label: "" sql_select: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) columns: [] # No default (required) where: "" # No default (optional) args_mapping: "" # No default (optional) ``` #### Advanced ```yml processors: label: "" sql_select: driver: "" # No default (required) dsn: "" # No default (required) table: "" # No default (required) columns: [] # No default (required) where: "" # No default (optional) args_mapping: "" # No default (optional) prefix: "" # No default (optional) suffix: "" # No default (optional) init_files: [] # No default (optional) init_statement: "" # No default (optional) conn_max_idle_time: "" # No default (optional) conn_max_life_time: "" # No default (optional) conn_max_idle: 2 conn_max_open: "" # No default (optional) ``` If the query fails to execute then the message will remain unchanged and the error can be caught using [error handling methods](../../../configuration/error_handling/). ## [](#examples)Examples ### [](#table-query-postgresql)Table Query (PostgreSQL) Here we query a database for columns of footable that share a `user_id` with the message `user.id`. A [`branch` processor](../branch/) is used in order to insert the resulting array into the original message at the path `foo_rows`: ```yaml pipeline: processors: - branch: processors: - sql_select: driver: postgres dsn: postgres://foouser:foopass@localhost:5432/testdb?sslmode=disable table: footable columns: [ '*' ] where: user_id = ? args_mapping: '[ this.user.id ]' result_map: 'root.foo_rows = this' ``` ## [](#fields)Fields ### [](#args_mapping)`args_mapping` An optional [Bloblang mapping](../../../guides/bloblang/about/) which should evaluate to an array of values matching in size to the number of placeholder arguments in the field `where`. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#columns)`columns[]` A list of columns to query. **Type**: `array` ```yaml # Examples: columns: - "*" # --- columns: - foo - bar - baz ``` ### [](#conn_max_idle)`conn_max_idle` An optional maximum number of connections in the idle connection pool. If conn\_max\_open is greater than 0 but less than the new conn\_max\_idle, then the new conn\_max\_idle will be reduced to match the conn\_max\_open limit. If `value ⇐ 0`, no idle connections are retained. The default max idle connections is currently 2. This may change in a future release. **Type**: `int` **Default**: `2` ### [](#conn_max_idle_time)`conn_max_idle_time` An optional maximum amount of time a connection may be idle. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections idle time. **Type**: `string` ### [](#conn_max_life_time)`conn_max_life_time` An optional maximum amount of time a connection may be reused. Expired connections may be closed lazily before reuse. If `value ⇐ 0`, connections are not closed due to a connections age. **Type**: `string` ### [](#conn_max_open)`conn_max_open` An optional maximum number of open connections to the database. If conn\_max\_idle is greater than 0 and the new conn\_max\_open is less than conn\_max\_idle, then conn\_max\_idle will be reduced to match the new conn\_max\_open limit. If `value ⇐ 0`, then there is no limit on the number of open connections. The default is 0 (unlimited). **Type**: `int` ### [](#driver)`driver` A database [driver](#drivers) to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks` ### [](#dsn)`dsn` A Data Source Name to identify the target database. #### [](#drivers)Drivers The following is a list of supported drivers, their placeholder style, and their respective DSN formats: | Driver | Data Source Name Format | | --- | --- | | clickhouse | clickhouse://[username[:password]@][netloc][:port]/dbname[?param1=value1&…​¶mN=valueN] | | mysql | [username[:password]@][protocol[(address)]]/dbname[?param1=value1&…​¶mN=valueN] | | postgres and pgx | postgres://[user[:password]@][netloc][:port][/dbname][?param1=value1&…​] | | mssql | sqlserver://[user[:password]@][netloc][:port][?database=dbname¶m1=value1&…​] | | sqlite | file:/path/to/filename.db[?param&=value1&…​] | | oracle | oracle://[username[:password]@][netloc][:port]/service_name?server=server2&server=server3 | | snowflake | username[:password]@account_identifier/dbname/schemaname[?param1=value&…​¶mN=valueN] | | trino | http[s]://user[:pass]@host[:port][?parameters] | | gocosmos | AccountEndpoint=;AccountKey=[;TimeoutMs=][;Version=][;DefaultDb/Db=][;AutoId=][;InsecureSkipVerify=] | | spanner | projects/[PROJECT]/instances/[INSTANCE]/databases/[DATABASE] | | databricks | token:@:/ | Please note that the `postgres` and `pgx` drivers enforce SSL by default, you can override this with the parameter `sslmode=disable` if required. The `pgx` driver is an alternative to the standard `postgres` (pq) driver and comes with extra functionality such as support for array insertion. The `snowflake` driver supports multiple DSN formats. Please consult [the docs](https://pkg.go.dev/github.com/snowflakedb/gosnowflake#hdr-Connection_String) for more details. For [key pair authentication](https://docs.snowflake.com/en/user-guide/key-pair-auth.html#configuring-key-pair-authentication), the DSN has the following format: `@//?warehouse=&role=&authenticator=snowflake_jwt&privateKey=`, where the value for the `privateKey` parameter can be constructed from an unencrypted RSA private key file `rsa_key.p8` using `openssl enc -d -base64 -in rsa_key.p8 | basenc --base64url -w0` (you can use `gbasenc` instead of `basenc` on OSX if you install `coreutils` via Homebrew). If you have a password-encrypted private key, you can decrypt it using `openssl pkcs8 -in rsa_key_encrypted.p8 -out rsa_key.p8`. Also, make sure fields such as the username are URL-encoded. The [`gocosmos`](https://pkg.go.dev/github.com/microsoft/gocosmos) driver is still experimental, but it has support for [hierarchical partition keys](https://learn.microsoft.com/en-us/azure/cosmos-db/hierarchical-partition-keys) as well as [cross-partition queries](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-query-container#cross-partition-query). Please refer to the [SQL notes](https://github.com/microsoft/gocosmos/blob/main/SQL.md) for details. **Type**: `string` ```yaml # Examples: dsn: clickhouse://username:password@host1:9000,host2:9000/database?dial_timeout=200ms&max_execution_time=60 # --- dsn: foouser:foopassword@tcp(localhost:3306)/foodb # --- dsn: postgres://foouser:foopass@localhost:5432/foodb?sslmode=disable # --- dsn: oracle://foouser:foopass@localhost:1521/service_name # --- dsn: token:dapi1234567890ab@dbc-a1b2345c-d6e7.cloud.databricks.com:443/sql/1.0/warehouses/abc123def456 ``` ### [](#init_files)`init_files[]` An optional list of file paths containing SQL statements to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Glob patterns are supported, including super globs (double star). Care should be taken to ensure that the statements are idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If a statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `array` ```yaml # Examples: init_files: - ./init/*.sql # --- init_files: - ./foo.sql - ./bar.sql ``` ### [](#init_statement)`init_statement` An optional SQL statement to execute immediately upon the first connection to the target database. This is a useful way to initialise tables before processing data. Care should be taken to ensure that the statement is idempotent, and therefore would not cause issues when run multiple times after service restarts. If both `init_statement` and `init_files` are specified the `init_statement` is executed _after_ the `init_files`. If the statement fails for any reason a warning log will be emitted but the operation of this component will not be stopped. Requires version 4.10.0 or later. **Type**: `string` ```yaml # Examples: init_statement: |- CREATE TABLE IF NOT EXISTS some_table ( foo varchar(50) not null, bar integer, baz varchar(50), primary key (foo) ) WITHOUT ROWID; ``` ### [](#prefix)`prefix` An optional prefix to prepend to the query (before SELECT). **Type**: `string` ### [](#suffix)`suffix` An optional suffix to append to the select query. **Type**: `string` ### [](#table)`table` The table to query. **Type**: `string` ```yaml # Examples: table: foo ``` ### [](#where)`where` An optional where clause to add. Placeholder arguments are populated with the `args_mapping` field. Placeholders should always be question marks, and will automatically be converted to dollar syntax when the postgres or clickhouse drivers are used. **Type**: `string` ```yaml # Examples: where: meow = ? and woof = ? # --- where: user_id = ? ``` --- # Page 293: sql **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/sql.md --- # sql --- title: sql latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/sql page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/sql.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/sql.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Processor ▼ [Processor](/redpanda-connect/components/processors/sql/)[Cache](/redpanda-connect/components/caches/sql/)[Output](/redpanda-connect/components/outputs/sql/) **Available in:** Self-Managed > ⚠️ **WARNING: Deprecated** > > Deprecated > > This component is deprecated and will be removed in the next major version release. Please consider moving onto [alternative components](#alternatives). Runs an arbitrary SQL query against a database and (optionally) returns the result as an array of objects, one for each row returned. Introduced in version 3.65.0. #### Common ```yml processors: label: "" sql: driver: "" # No default (required) data_source_name: "" # No default (required) query: "" # No default (required) args_mapping: "" # No default (optional) result_codec: none ``` #### Advanced ```yml processors: label: "" sql: driver: "" # No default (required) data_source_name: "" # No default (required) query: "" # No default (required) unsafe_dynamic_query: false args_mapping: "" # No default (optional) result_codec: none ``` If the query fails to execute then the message will remain unchanged and the error can be caught using [error handling methods](../../../configuration/error_handling/). ## [](#alternatives)Alternatives For basic inserts or select queries use either the [`sql_insert`](../sql_insert/) or the [`sql_select`](../sql_select/) processor. For more complex queries use the [`sql_raw`](../sql_raw/) processor. ## [](#fields)Fields ### [](#args_mapping)`args_mapping` An optional [Bloblang mapping](../../../guides/bloblang/about/) which should evaluate to an array of values matching in size to the number of placeholder arguments in the field `query`. **Type**: `string` ```yaml # Examples: args_mapping: root = [ this.cat.meow, this.doc.woofs[0] ] # --- args_mapping: root = [ meta("user.id") ] ``` ### [](#data_source_name)`data_source_name` Data source name. **Type**: `string` ### [](#driver)`driver` A database [driver](#drivers) to use. **Type**: `string` **Options**: `mysql`, `postgres`, `pgx`, `clickhouse`, `mssql`, `sqlite`, `oracle`, `snowflake`, `trino`, `gocosmos`, `spanner`, `databricks` ### [](#query)`query` The query to execute. The style of placeholder to use depends on the driver, some drivers require question marks (`?`) whereas others expect incrementing dollar signs (`$1`, `$2`, and so on) or colons (`:1`, `:2` and so on). The style to use is outlined in this table: | Driver | Placeholder Style | | --- | --- | | clickhouse | Dollar sign ($) | | gocosmos | Colon (:) | | mysql | Question mark (?) | | mssql | Question mark (?) | | oracle | Colon (:) | | postgres | Dollar sign ($) | | snowflake | Question mark (?) | | spanner | Question mark (?) | | sqlite | Question mark (?) | | trino | Question mark (?) | **Type**: `string` ```yaml # Examples: query: INSERT INTO footable (foo, bar, baz) VALUES (?, ?, ?); ``` ### [](#result_codec)`result_codec` Result codec. **Type**: `string` **Default**: `none` ### [](#unsafe_dynamic_query)`unsafe_dynamic_query` Whether to enable [interpolation functions](../../../configuration/interpolation/#bloblang-queries) in the query. Great care should be made to ensure your queries are defended against injection attacks. **Type**: `bool` **Default**: `false` --- # Page 294: string_split **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/string_split.md --- # string\_split --- title: string_split latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/string_split page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/string_split.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/string_split.adoc description: Splits a string by a delimiter into an array. Generally, using bloblang's `split` method is preferred. In some high performance use cases this processor can be faster than the equivalent bloblang if there is no additional logic. page-git-created-date: "2026-04-03" page-git-modified-date: "2026-04-03" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/string_split/ "View the Cloud version of this component"), Self-Managed Splits a string by a delimiter into an array. Generally, using bloblang’s `split` method is preferred. In some high performance use cases this processor can be faster than the equivalent bloblang if there is no additional logic. #### Common ```yml processors: label: "" string_split: delimiter: empty_as_null: false ``` #### Advanced ```yml processors: label: "" string_split: delimiter: emit_bytes: false empty_as_null: false ``` ## [](#fields)Fields ### [](#delimiter)`delimiter` The delimiter to split the string by. **Type**: `string` **Default**: \` \` ### [](#emit_bytes)`emit_bytes` When true, the output will be bloblang bytes instead of strings. **Type**: `bool` **Default**: `false` ### [](#empty_as_null)`empty_as_null` When true, empty strings resulting from the split are converted to null. **Type**: `bool` **Default**: `false` --- # Page 295: subprocess **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/subprocess.md --- # subprocess --- title: subprocess latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/subprocess page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/subprocess.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/subprocess.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Processor ▼ [Processor](/redpanda-connect/components/processors/subprocess/)[Input](/redpanda-connect/components/inputs/subprocess/)[Output](/redpanda-connect/components/outputs/subprocess/) **Available in:** Self-Managed Executes a command as a subprocess and, for each message, will pipe its contents to the stdin stream of the process followed by a newline. #### Common ```yml processors: label: "" subprocess: name: "" # No default (required) args: [] ``` #### Advanced ```yml processors: label: "" subprocess: name: "" # No default (required) args: [] max_buffer: 65536 codec_send: lines codec_recv: lines ``` > 📝 **NOTE** > > This processor keeps the subprocess alive and requires very specific behavior from the command executed. If you wish to simply execute a command for each message take a look at the [`command` processor](../command/) instead. The subprocess must then either return a line over stdout or stderr. If a response is returned over stdout then its contents will replace the message. If a response is instead returned from stderr it will be logged and the message will continue unchanged and will be [marked as failed](../../../configuration/error_handling/). Rather than separating data by a newline it’s possible to specify alternative [`codec_send`](#codec_send) and [`codec_recv`](#codec_recv) values, which allow binary messages to be encoded for logical separation. The execution environment of the subprocess is the same as the Redpanda Connect instance, including environment variables and the current working directory. The field `max_buffer` defines the maximum response size able to be read from the subprocess. This value should be set significantly above the real expected maximum response size. ## [](#subprocess-requirements)Subprocess requirements It is required that subprocesses flush their stdout and stderr pipes for each line. Redpanda Connect will attempt to keep the process alive for as long as the pipeline is running. If the process exits early it will be restarted. ## [](#messages-containing-line-breaks)Messages containing line breaks If a message contains line breaks each line of the message is piped to the subprocess and flushed, and a response is expected from the subprocess before another line is fed in. ## [](#fields)Fields ### [](#args)`args[]` A list of arguments to provide the command. **Type**: `array` **Default**: `[]` ### [](#codec_recv)`codec_recv` Determines how messages read from the subprocess are decoded, which allows them to be logically separated. Requires version 3.37.0 or later. **Type**: `string` **Default**: `lines` **Options**: `lines`, `length_prefixed_uint32_be`, `netstring` ### [](#codec_send)`codec_send` Determines how messages written to the subprocess are encoded, which allows them to be logically separated. Requires version 3.37.0 or later. **Type**: `string` **Default**: `lines` **Options**: `lines`, `length_prefixed_uint32_be`, `netstring` ### [](#max_buffer)`max_buffer` The maximum expected response size. **Type**: `int` **Default**: `65536` ### [](#name)`name` The command to execute as a subprocess. **Type**: `string` ```yaml # Examples: name: cat # --- name: sed # --- name: awk ``` --- # Page 296: switch **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/switch.md --- # switch --- title: switch latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/switch page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/switch.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/switch.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Type:** Processor ▼ [Processor](/redpanda-connect/components/processors/switch/)[Output](/redpanda-connect/components/outputs/switch/)[Scanner](/redpanda-connect/components/scanners/switch/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/switch/ "View the Cloud version of this component"), Self-Managed Conditionally processes messages based on their contents. ```yml # Config fields, showing default values label: "" switch: [] # No default (required) ``` For each switch case a [Bloblang query](../../../guides/bloblang/about/) is checked and, if the result is true (or the check is empty) the child processors are executed on the message. ## [](#fields)Fields ### [](#check)`check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether a message should have the processors of this case executed on it. If left empty the case always passes. If the check mapping throws an error the message will be flagged [as having failed](../../../configuration/error_handling/) and will not be tested against any other cases. **Type**: `string` **Default**: `""` ```yaml # Examples: check: this.type == "foo" # --- check: this.contents.urls.contains("https://benthos.dev/") ``` ### [](#continue)`continue` Indicates whether, if this case passes for a message, the next case should also be tested. Unlike `fallthrough`, which skips the next case’s check, `continue` will evaluate the next case’s condition before executing. **Type**: `bool` **Default**: `false` ### [](#fallthrough)`fallthrough` Indicates whether, if this case passes for a message, the next case should also be executed without checking its condition. **Type**: `bool` **Default**: `false` ### [](#processors)`processors[]` A list of [processors](../about/) to execute on a message. **Type**: `processor` **Default**: `[]` ## [](#examples)Examples ### [](#ignore-george)Ignore George We have a system where we’re counting a metric for all messages that pass through our system. However, occasionally we get messages from George that we don’t care about. For George’s messages we want to instead emit a metric that gauges how angry he is about being ignored and then we drop it. ```yaml pipeline: processors: - switch: - check: this.user.name.first != "George" processors: - metric: type: counter name: MessagesWeCareAbout - processors: - metric: type: gauge name: GeorgesAnger value: ${! json("user.anger") } - mapping: root = deleted() ``` ## [](#batching)Batching When a switch processor executes on a [batch of messages](../../../configuration/batching/) they are checked individually and can be matched independently against cases. During processing the messages matched against a case are processed as a batch, although the ordering of messages during case processing cannot be guaranteed to match the order as received. At the end of switch processing the resulting batch will follow the same ordering as the batch was received. If any child processors have split or otherwise grouped messages this grouping will be lost as the result of a switch is always a single batch. In order to perform conditional grouping and/or splitting use the [`group_by` processor](../group_by/). --- # Page 297: sync_response **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/sync_response.md --- # sync\_response --- title: sync_response latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/sync_response page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/sync_response.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/sync_response.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-09" --- **Type:** Processor ▼ [Processor](/redpanda-connect/components/processors/sync_response/)[Output](/redpanda-connect/components/outputs/sync_response/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/sync_response/ "View the Cloud version of this component"), Self-Managed Adds the payload in its current state as a synchronous response to the input source, where it is dealt with according to that specific input type. ```yml # Config fields, showing default values label: "" sync_response: {} ``` For most inputs this mechanism is ignored entirely, in which case the sync response is dropped without penalty. It is therefore safe to use this processor even when combining input types that might not have support for sync responses. An example of an input able to utilize this is the `http_server`. For more information please read [synchronous responses](../../../guides/sync_responses/). --- # Page 298: text_chunker **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/text_chunker.md --- # text\_chunker --- title: text_chunker latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/text_chunker page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/text_chunker.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/text_chunker.adoc page-git-created-date: "2025-05-02" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/text_chunker/ "View the Cloud version of this component"), Self-Managed Breaks down text-based message content into manageable chunks using a configurable strategy. This processor is ideal for creating vector embeddings of large text documents. Introduced in version 4.51.0. #### Common ```yml processors: label: "" text_chunker: strategy: "" # No default (required) chunk_size: 512 chunk_overlap: 100 separators: - "\n\n" - "\n" - " " - "" length_measure: runes include_code_blocks: false keep_reference_links: false ``` #### Advanced ```yml processors: label: "" text_chunker: strategy: "" # No default (required) chunk_size: 512 chunk_overlap: 100 separators: - "\n\n" - "\n" - " " - "" length_measure: runes token_encoding: "" # No default (optional) allowed_special: [] disallowed_special: - "all" include_code_blocks: false keep_reference_links: false ``` ## [](#fields)Fields ### [](#allowed_special)`allowed_special[]` A list of special tokens to include in the output from this processor. **Type**: `array` **Default**: `[]` ### [](#chunk_overlap)`chunk_overlap` The number of characters duplicated in adjacent chunks of text. **Type**: `int` **Default**: `100` ### [](#chunk_size)`chunk_size` The maximum size of each chunk, using the selected [`length_measure`](#length_measure). **Type**: `int` **Default**: `512` ### [](#disallowed_special)`disallowed_special[]` A list of special tokens to exclude from the output of this processor. **Type**: `array` **Default**: ```yaml - "all" ``` ### [](#include_code_blocks)`include_code_blocks` When set to `true`, this processor includes code blocks in the output. **Type**: `bool` **Default**: `false` ### [](#keep_reference_links)`keep_reference_links` When set to `true`, this processor includes reference links in the output. **Type**: `bool` **Default**: `false` ### [](#length_measure)`length_measure` Choose a method to measure the length of a string. **Type**: `string` **Default**: `runes` | Option | Summary | | --- | --- | | graphemes | Use unicode graphemes to determine the length of a string. | | runes | Use the number of codepoints to determine the length of a string. | | token | Use the number of tokens (using the token_encoding tokenizer) to determine the length of a string. | | utf8 | Determine the length of text using the number of utf8 bytes. | ### [](#separators)`separators[]` A list of strings to use as separators between chunks when the [`recursive_character` strategy option](#strategy) is specified. By default, the following separators are tried in turn until one is successful: - Double newlines (\` `) - Single newlines (` ``) - Spaces (`" “,”"``) **Type**: `array` **Default**: ```yaml - "\n\n" - "\n" - " " - "" ``` ### [](#strategy)`strategy` Choose a strategy for breaking content down into chunks. **Type**: `string` | Option | Summary | | --- | --- | | markdown | Split text by markdown headers. | | recursive_character | Split text recursively by characters (defined in separators). | | token | Split text by tokens. | ### [](#token_encoding)`token_encoding` The type of encoding to use for tokenization. **Type**: `string` ```yaml # Examples: token_encoding: cl100k_base # --- token_encoding: r50k_base ``` --- # Page 299: try **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/try.md --- # try --- title: try latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/try page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/try.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/try.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/try/ "View the Cloud version of this component"), Self-Managed Executes a list of child processors on messages only if no prior processors have failed (or the errors have been cleared). ```yml # Config fields, showing default values label: "" try: [] ``` This processor behaves similarly to the [`for_each`](../for_each/) processor, where a list of child processors are applied to individual messages of a batch. However, if a message has failed any prior processor (before or during the try block) then that message will skip all following processors. For example, with the following config: ```yaml pipeline: processors: - resource: foo - try: - resource: bar - resource: baz - resource: buz ``` If the processor `bar` fails for a particular message, that message will skip the processors `baz` and `buz`. Similarly, if `bar` succeeds but `baz` does not then `buz` will be skipped. If the processor `foo` fails for a message then none of `bar`, `baz` or `buz` are executed on that message. This processor is useful for when child processors depend on the successful output of previous processors. This processor can be followed with a [catch](../catch/) processor for defining child processors to be applied only to failed messages. More information about error handing can be found in [Error Handling](../../../configuration/error_handling/). ## [](#nest-within-a-catch-block)Nest within a catch block In some cases it might be useful to nest a try block within a catch block, since the [`catch` processor](../catch/) only clears errors _after_ executing its child processors this means a nested try processor will not execute unless the errors are explicitly cleared beforehand. This can be done by inserting an empty catch block before the try block like as follows: ```yaml pipeline: processors: - resource: foo - catch: - log: level: ERROR message: "Foo failed due to: ${! error() }" - catch: [] # Clear prior error - try: - resource: bar - resource: baz ``` --- # Page 300: unarchive **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/unarchive.md --- # unarchive --- title: unarchive latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/unarchive page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/unarchive.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/unarchive.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-13" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/unarchive/ "View the Cloud version of this component"), Self-Managed Unarchives messages according to the selected archive format into multiple messages within a [batch](../../../configuration/batching/). ```yml # Config fields, showing default values label: "" unarchive: format: "" # No default (required) ``` When a message is unarchived the new messages replace the original message in the batch. Messages that are selected but fail to unarchive (invalid format) will remain unchanged in the message batch but will be flagged as having failed, allowing you to [error handle them](../../../configuration/error_handling/). ## [](#metadata)Metadata The metadata found on the messages handled by this processor will be copied into the resulting messages. For the unarchive formats that contain file information (tar, zip), a metadata field is also added to each message called `archive_filename` with the extracted filename. ## [](#fields)Fields ### [](#format)`format` The unarchiving format to apply. **Type**: `string` | Option | Summary | | --- | --- | | binary | Extract messages from a binary blob format. | | csv | Attempt to parse the message as a csv file (header required) and for each row in the file expands its contents into a json object in a new message. | | csv:x | Attempt to parse the message as a csv file (header required) and for each row in the file expands its contents into a json object in a new message using a custom delimiter. The custom delimiter must be a single character, e.g. the format "csv:\t" would consume a tab delimited file. | | json_array | Attempt to parse a message as a JSON array, and extract each element into its own message. | | json_documents | Attempt to parse a message as a stream of concatenated JSON documents. Each parsed document is expanded into a new message. | | json_map | Attempt to parse the message as a JSON map and for each element of the map expands its contents into a new message. A metadata field is added to each message called archive_key with the relevant key from the top-level map. | | lines | Extract the lines of a message each into their own message. | | tar | Extract messages from a unix standard tape archive. | | zip | Extract messages from a zip file. | --- # Page 301: wasm **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/wasm.md --- # wasm --- title: wasm latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/wasm page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/wasm.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/wasm.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Available in:** Self-Managed Executes a function exported by a WASM module for each message. Introduced in version 4.11.0. ```yml # Config fields, showing default values label: "" wasm: module_path: "" # No default (required) function: process ``` This processor uses [Wazero](https://github.com/tetratelabs/wazero) to execute a WASM module (with support for WASI), calling a specific function for each message being processed. From within the WASM module it is possible to query and mutate the message being processed via a suite of functions exported to the module. This ecosystem is delicate as WASM doesn’t have a single clearly defined way to pass strings back and forth between the host and the module. In order to remedy this we’re gradually working on introducing libraries and examples for multiple languages which can be found in [the codebase](https://github.com/redpanda-data/benthos/tree/main/public/wasm/README.md). These examples, as well as the processor itself, is a work in progress. ## [](#parallelism)Parallelism It’s not currently possible to execute a single WASM runtime across parallel threads with this processor. Therefore, in order to support parallel processing this processor implements pooling of module runtimes. Ideally your WASM module shouldn’t depend on any global state, but if it does then you need to ensure the processor [is only run on a single thread](../../../configuration/processing_pipelines/). ## [](#fields)Fields ### [](#function)`function` The name of the function exported by the target WASM module to run for each message. **Type**: `string` **Default**: `process` ### [](#module_path)`module_path` The path of the target WASM module to execute. **Type**: `string` --- # Page 302: while **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/while.md --- # while --- title: while latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/while page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/while.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/while.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/while/ "View the Cloud version of this component"), Self-Managed A processor that checks a [Bloblang query](../../../guides/bloblang/about/) against each batch of messages and executes child processors on them for as long as the query resolves to true. #### Common ```yml processors: label: "" while: at_least_once: false check: "" processors: [] # No default (required) ``` #### Advanced ```yml processors: label: "" while: at_least_once: false max_loops: 0 check: "" processors: [] # No default (required) ``` The field `at_least_once`, if true, ensures that the child processors are always executed at least one time (like a do .. while loop.) The field `max_loops`, if greater than zero, caps the number of loops for a message batch to this value. If following a loop execution the number of messages in a batch is reduced to zero the loop is exited regardless of the condition result. If following a loop execution there are more than 1 message batches the query is checked against the first batch only. The conditions of this processor are applied across entire message batches. You can find out more about batching [in this doc](../../../configuration/batching/). ## [](#fields)Fields ### [](#at_least_once)`at_least_once` Whether to always run the child processors at least one time. **Type**: `bool` **Default**: `false` ### [](#check)`check` A [Bloblang query](../../../guides/bloblang/about/) that should return a boolean value indicating whether the while loop should execute again. **Type**: `string` **Default**: `""` ```yaml # Examples: check: errored() # --- check: this.urls.unprocessed.length() > 0 ``` ### [](#max_loops)`max_loops` An optional maximum number of loops to execute. Helps protect against accidentally creating infinite loops. **Type**: `int` **Default**: `0` ### [](#processors)`processors[]` A list of child processors to execute on each loop. **Type**: `processor` --- # Page 303: workflow **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/workflow.md --- # workflow --- title: workflow latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/workflow page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/workflow.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/workflow.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/workflow/ "View the Cloud version of this component"), Self-Managed Executes a topology of [`branch` processors](../branch/), performing them in parallel where possible. #### Common ```yml processors: label: "" workflow: meta_path: meta.workflow order: [] branches: request_map: "" processors: [] # No default (required) result_map: "" ``` #### Advanced ```yml processors: label: "" workflow: meta_path: meta.workflow order: [] branch_resources: [] branches: request_map: "" processors: [] # No default (required) result_map: "" ``` ## [](#why-use-a-workflow)Why use a workflow ### [](#performance)Performance Most of the time the best way to compose processors is also the simplest, just configure them in series. This is because processors are often CPU bound, low-latency, and you can gain vertical scaling by increasing the number of processor pipeline threads, allowing Redpanda Connect to process [multiple messages in parallel](../../../configuration/processing_pipelines/). However, some processors, such as [`http`](../http/), [`aws_lambda`](../aws_lambda/) or [`cache`](../cache/), interact with external services and therefore spend most of their time waiting for a response. These processors tend to be high-latency and low CPU activity, which causes messages to process slowly. When a processing pipeline contains multiple network processors that aren’t dependent on each other we can benefit from performing these processors in parallel for each individual message, reducing the overall message processing latency. ### [](#simplifying-processor-topology)Simplifying processor topology A workflow is often expressed as a [DAG](https://en.wikipedia.org/wiki/Directed_acyclic_graph) of processing stages, where each stage can result in N possible next stages, until finally the flow ends at an exit node. For example, if we had processing stages A, B, C and D, where stage A could result in either stage B or C being next, always followed by D, it might look something like this: ```text /--> B --\ A --| |--> D \--> C --/ ``` This flow would be easy to express in a standard Redpanda Connect config, we could simply use a [`switch` processor](../switch/) to route to either B or C depending on a condition on the result of A. However, this method of flow control quickly becomes unfeasible as the DAG gets more complicated, imagine expressing this flow using switch processors: ```text /--> B -------------|--> D / / A --| /--> E --| \--> C --| \ \----------|--> F ``` And imagine doing so knowing that the diagram is subject to change over time. Yikes! Instead, with a workflow we can either trust it to automatically resolve the DAG or express it manually as simply as `order: [ [ A ], [ B, C ], [ E ], [ D, F ] ]`, and the conditional logic for determining if a stage is executed is defined as part of the branch itself. ## [](#examples)Examples ### [](#automatic-ordering)Automatic Ordering When the field `order` is omitted a best attempt is made to determine a dependency tree between branches based on their request and result mappings. In the following example the branches foo and bar will be executed first in parallel, and afterwards the branch baz will be executed. ```yaml pipeline: processors: - workflow: meta_path: meta.workflow branches: foo: request_map: 'root = ""' processors: - http: url: TODO result_map: 'root.foo = this' bar: request_map: 'root = this.body' processors: - aws_lambda: function: TODO result_map: 'root.bar = this' baz: request_map: | root.fooid = this.foo.id root.barstuff = this.bar.content processors: - cache: resource: TODO operator: set key: ${! json("fooid") } value: ${! json("barstuff") } ``` ### [](#conditional-branches)Conditional Branches Branches of a workflow are skipped when the `request_map` assigns `deleted()` to the root. In this example the branch A is executed when the document type is "foo", and branch B otherwise. Branch C is executed afterwards and is skipped unless either A or B successfully provided a result at `tmp.result`. ```yaml pipeline: processors: - workflow: branches: A: request_map: | root = if this.document.type != "foo" { deleted() } processors: - http: url: TODO result_map: 'root.tmp.result = this' B: request_map: | root = if this.document.type == "foo" { deleted() } processors: - aws_lambda: function: TODO result_map: 'root.tmp.result = this' C: request_map: | root = if this.tmp.result != null { deleted() } processors: - http: url: TODO_SOMEWHERE_ELSE result_map: 'root.tmp.result = this' ``` ### [](#resources)Resources The `order` field can be used in order to refer to [branch processor resources](#resources), this can sometimes make your pipeline configuration cleaner, as well as allowing you to reuse branch configurations in order places. It’s also possible to mix and match branches configured within the workflow and configured as resources. ```yaml pipeline: processors: - workflow: order: [ [ foo, bar ], [ baz ] ] branches: bar: request_map: 'root = this.body' processors: - aws_lambda: function: TODO result_map: 'root.bar = this' processor_resources: - label: foo branch: request_map: 'root = ""' processors: - http: url: TODO result_map: 'root.foo = this' - label: baz branch: request_map: | root.fooid = this.foo.id root.barstuff = this.bar.content processors: - cache: resource: TODO operator: set key: ${! json("fooid") } value: ${! json("barstuff") } ``` ## [](#fields)Fields ### [](#branch_resources)`branch_resources[]` An optional list of [`branch` processor](../branch/) names that are configured as [Resources](#resources). These resources will be included in the workflow with any branches configured inline within the [`branches`](#branches) field. The order and parallelism in which branches are executed is automatically resolved based on the mappings of each branch. When using resources with an explicit order it is not necessary to list resources in this field. Requires version 3.38.0 or later. **Type**: `array` **Default**: `[]` ### [](#branches)`branches` An object of named [`branch` processors](../branch/) that make up the workflow. The order and parallelism in which branches are executed can either be made explicit with the field `order`, or if omitted an attempt is made to automatically resolve an ordering based on the mappings of each branch. **Type**: `object` **Default**: `{}` ### [](#branches-processors)`branches.processors[]` A list of processors to apply to mapped requests. When processing message batches the resulting batch must match the size and ordering of the input batch, therefore filtering, grouping should not be performed within these processors. **Type**: `processor` ### [](#branches-request_map)`branches.request_map` A [Bloblang mapping](../../../guides/bloblang/about/) that describes how to create a request payload suitable for the child processors of this branch. If left empty then the branch will begin with an exact copy of the origin message (including metadata). **Type**: `string` **Default**: `""` ```yaml # Examples: request_map: |- root = { "id": this.doc.id, "content": this.doc.body.text } # --- request_map: |- root = if this.type == "foo" { this.foo.request } else { deleted() } ``` ### [](#branches-result_map)`branches.result_map` A [Bloblang mapping](../../../guides/bloblang/about/) that describes how the resulting messages from branched processing should be mapped back into the original payload. If left empty the origin message will remain unchanged (including metadata). **Type**: `string` **Default**: `""` ```yaml # Examples: result_map: |- meta foo_code = metadata("code") root.foo_result = this # --- result_map: |- meta = metadata() root.bar.body = this.body root.bar.id = this.user.id # --- result_map: root.raw_result = content().string() # --- result_map: |- root.enrichments.foo = if metadata("request_failed") != null { throw(metadata("request_failed")) } else { this } # --- result_map: |- # Retain only the updated metadata fields which were present in the origin message meta = metadata().filter(v -> @.get(v.key) != null) ``` ### [](#meta_path)`meta_path` A [dot path](../../../configuration/field_paths/) indicating where to store and reference [structured metadata](#structured-metadata) about the workflow execution. **Type**: `string` **Default**: `meta.workflow` ### [](#order)`order` An explicit declaration of branch ordered tiers, which describes the order in which parallel tiers of branches should be executed. Branches should be identified by the name as they are configured in the field `branches`. It’s also possible to specify branch processors configured [as a resource](#resources). **Type**: `string` **Default**: `[]` ```yaml # Examples: order: - - foo - bar - - baz # --- order: - - foo - - bar - - baz ``` ## [](#structured-metadata)Structured metadata When the field `meta_path` is non-empty the workflow processor creates an object describing which workflows were successful, skipped or failed for each message and stores the object within the message at the end. The object is of the following form: ```json { "succeeded": [ "foo" ], "skipped": [ "bar" ], "failed": { "baz": "the error message from the branch" } } ``` If a message already has a meta object at the given path when it is processed then the object is used in order to determine which branches have already been performed on the message (or skipped) and can therefore be skipped on this run. This is a useful pattern when replaying messages that have failed some branches previously. For example, given the above example object the branches foo and bar would automatically be skipped, and baz would be reattempted. The previous meta object will also be preserved in the field `.previous` when the new meta object is written, preserving a full record of all workflow executions. If a field `.apply` exists in the meta object for a message and is an array then it will be used as an explicit list of stages to apply, all other stages will be skipped. ## [](#resources-2)Resources It’s common to configure processors (and other components) [as resources](../../../configuration/resources/) in order to keep the pipeline configuration cleaner. With the workflow processor you can include branch processors configured as resources within your workflow either by specifying them by name in the field `order`, if Redpanda Connect doesn’t find a branch within the workflow configuration of that name it’ll refer to the resources. Alternatively, if you do not wish to have an explicit ordering, you can add resource names to the field `branch_resources` and they will be included in the workflow with automatic DAG resolution along with any branches configured in the `branches` field. ### [](#resource-error-conditions)Resource error conditions There are two error conditions that could potentially occur when resources included in your workflow are mutated, and if you are planning to mutate resources in your workflow it is important that you understand them. The first error case is that a resource in the workflow is removed and not replaced, when this happens the workflow will still be executed but the individual branch will fail. This should only happen if you explicitly delete a branch resource, as any mutation operation will create the new resource before removing the old one. The second error case is when automatic DAG resolution is being used and a resource in the workflow is changed in a way that breaks the DAG (circular dependencies, etc). When this happens it is impossible to execute the workflow and therefore the processor will fail, which is possible to capture and handle using [standard error handling patterns](../../../configuration/error_handling/). ## [](#error-handling)Error handling The recommended approach to handle failures within a workflow is to query against the [structured metadata](#structured-metadata) it provides, as it provides granular information about exactly which branches failed and which ones succeeded and therefore aren’t necessary to perform again. For example, if our meta object is stored at the path `meta.workflow` and we wanted to check whether a message has failed for any branch we can do that using a [Bloblang query](../../../guides/bloblang/about/) like `this.meta.workflow.failed.length() | 0 > 0`, or to check whether a specific branch failed we can use `this.exists("meta.workflow.failed.foo")`. However, if structured metadata is disabled by setting the field `meta_path` to empty then the workflow processor instead adds a general error flag to messages when any executed branch fails. In this case it’s possible to handle failures using [standard error handling patterns](../../../configuration/error_handling/). --- # Page 304: xml **URL**: https://docs.redpanda.com/redpanda-connect/components/processors/xml.md --- # xml --- title: xml latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processors/xml page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processors/xml.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/processors/xml.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-03-10" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/processors/xml/ "View the Cloud version of this component"), Self-Managed Parses messages as an XML document, performs a mutation on the data, and then overwrites the previous contents with the new value. ```yml # Config fields, showing default values label: "" xml: operator: "" cast: false ``` ## [](#operators)Operators ### [](#to_json)`to_json` Converts an XML document into a JSON structure, where elements appear as keys of an object according to the following rules: - If an element contains attributes they are parsed by prefixing a hyphen, `-`, to the attribute label. - If the element is a simple element and has attributes, the element value is given the key `#text`. - XML comments, directives, and process instructions are ignored. - When elements are repeated the resulting JSON value is an array. - XML namespaces are stripped from element and attribute names, and namespace declarations (`xmlns`) are omitted. For example, given the following XML: ```xml This is a title This is a description foo1 foo2 foo3 ``` The resulting JSON structure would look like this: ```json { "root":{ "title":"This is a title", "description":{ "#text":"This is a description", "-tone":"boring" }, "elements":[ {"#text":"foo1","-id":"1"}, {"#text":"foo2","-id":"2"}, "foo3" ] } } ``` With cast set to true, the resulting JSON structure would look like this: ```json { "root":{ "title":"This is a title", "description":{ "#text":"This is a description", "-tone":"boring" }, "elements":[ {"#text":"foo1","-id":1}, {"#text":"foo2","-id":2}, "foo3" ] } } ``` ## [](#fields)Fields ### [](#cast)`cast` Whether to try to cast values that are numbers and booleans to the right type. Default: all values are strings. **Type**: `bool` **Default**: `false` ### [](#operator)`operator` An XML [operation](#operators) to apply to messages. **Type**: `string` **Default**: `""` **Options**: `to_json` --- # Page 305: Rate Limits **URL**: https://docs.redpanda.com/redpanda-connect/components/rate_limits/about.md --- # Rate Limits --- title: Rate Limits latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: rate_limits/about page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: rate_limits/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/rate_limits/about.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-09" --- A rate limit is a strategy for limiting the usage of a shared resource across parallel components in a Redpanda Connect instance, or potentially across multiple instances. They are configured as a resource: ```yaml rate_limit_resources: - label: foobar local: count: 500 interval: 1s ``` And most components that hit external services have a field `rate_limit` for specifying a rate limit resource to use, identified by the `label` field. For example, if we wanted to use our `foobar` rate limit with an [`http_client`](../../inputs/http_client/) input it would look like this: ```yaml input: http_client: url: TODO verb: GET rate_limit: foobar ``` By using a rate limit in this way we can guarantee that our input will only poll our HTTP source at the rate of 500 requests per second. Some components don’t have a `rate_limit` field but we might still wish to throttle them by a rate limit, in which case we can use the [`rate_limit` processor](../../processors/rate_limit/) that applies back pressure to a processing pipeline when the limit is reached. For example, if we wished to limit the consumption of lines of a [`csv` file input](../../inputs/csv/) to a specified rate limit we can do that with the following: ```yaml input: csv: paths: - ./foo.csv processors: - rate_limit: resource: foobar ``` You can find out more about resources [in this document.](../../../configuration/resources/) --- # Page 306: local **URL**: https://docs.redpanda.com/redpanda-connect/components/rate_limits/local.md --- # local --- title: local latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: rate_limits/local page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: rate_limits/local.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/rate_limits/local.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/rate_limits/local/ "View the Cloud version of this component"), Self-Managed The local rate limit is a simple X every Y type rate limit that can be shared across any number of components within the pipeline but does not support distributed rate limits across multiple running instances of Benthos. ```yml # Config fields, showing default values label: "" local: count: 1000 interval: 1s ``` ## [](#fields)Fields ### [](#count)`count` The maximum number of requests to allow for a given period of time. **Type**: `int` **Default**: `1000` ### [](#interval)`interval` The time window to limit requests by. **Type**: `string` **Default**: `"1s"` --- # Page 307: redis **URL**: https://docs.redpanda.com/redpanda-connect/components/rate_limits/redis.md --- # redis --- title: redis latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: rate_limits/redis page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: rate_limits/redis.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/rate_limits/redis.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- **Type:** Rate\_limit ▼ [Rate\_limit](/redpanda-connect/components/rate_limits/redis/)[Cache](/redpanda-connect/components/caches/redis/)[Processor](/redpanda-connect/components/processors/redis/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/rate_limits/redis/ "View the Cloud version of this component"), Self-Managed A rate limit implementation using Redis. It works by using a simple token bucket algorithm to limit the number of requests to a given count within a given time period. The rate limit is shared across all instances of Redpanda Connect that use the same Redis instance, which must all have a consistent count and interval. Introduced in version 4.12.0. #### Common ```yml # Common config fields, showing default values label: "" redis: url: redis://:6379 # No default (required) count: 1000 interval: 1s key: "" # No default (required) ``` #### Advanced ```yml # All config fields, showing default values label: "" redis: url: redis://:6379 # No default (required) kind: simple master: "" tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] count: 1000 interval: 1s key: "" # No default (required) ``` ## [](#fields)Fields ### [](#url)`url` The URL of the target Redis server. Database is optional and is supplied as the URL path. **Type**: `string` ```yml # Examples url: redis://:6379 url: redis://localhost:6379 url: redis://foousername:foopassword@redisplace:6379 url: redis://:foopassword@redisplace:6379 url: redis://localhost:6379/1 url: redis://localhost:6379/1,redis://localhost:6380/1 ``` ### [](#kind)`kind` Specifies a simple, cluster-aware, or failover-aware redis client. **Type**: `string` **Default**: `"simple"` Options: `simple` , `cluster` , `failover` . ### [](#master)`master` Name of the redis master when `kind` is `failover` **Type**: `string` **Default**: `""` ```yml # Examples master: mymaster ``` ### [](#tls)`tls` Custom TLS settings can be used to override system defaults. **Troubleshooting** Some cloud hosted instances of Redis (such as Azure Cache) might need some hand holding in order to establish stable connections. Unfortunately, it is often the case that TLS issues will manifest as generic error messages such as "i/o timeout". If you’re using TLS and are seeing connectivity problems consider setting `enable_renegotiation` to `true`, and ensuring that the server supports at least TLS version 1.2. **Type**: `object` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` Requires version 3.45.0 or newer ### [](#tls-root_cas)`tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yml # Examples root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yml # Examples root_cas_file: ./root_cas.pem ``` ### [](#tls-client_certs)`tls.client_certs` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `array` **Default**: `[]` ```yml # Examples client_certs: - cert: foo key: bar client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yml # Examples password: foo password: ${KEY_PASSWORD} ``` ### [](#count)`count` The maximum number of messages to allow for a given period of time. **Type**: `int` **Default**: `1000` ### [](#interval)`interval` The time window to limit requests by. **Type**: `string` **Default**: `"1s"` ### [](#key)`key` The key to use for the rate limit. **Type**: `string` --- # Page 308: redpanda **URL**: https://docs.redpanda.com/redpanda-connect/components/redpanda/about.md --- # redpanda --- title: redpanda latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: redpanda/about page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: redpanda/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/redpanda/about.adoc page-git-created-date: "2024-05-29" page-git-modified-date: "2025-07-08" --- > 📝 **NOTE** > > This component requires an [enterprise license](../../../get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that’s valid for 30 days. The Redpanda Connect configuration service allows you to: - Configure Redpanda cluster credentials in a single configuration block, which is referenced by multiple components in data pipeline. For more information, see the [Pipeline example](#pipeline-example). - Send logs and status updates to topics on a Redpanda cluster, in addition to the [default logger](../../logger/about/). The `redpanda` namespace contains the configuration of this service. #### Common ```yml # Common configuration fields, showing default values redpanda: seed_brokers: [] # No default (optional) pipeline_id: "" logs_topic: "" logs_level: info status_topic: "" ``` #### Advanced ```yml # All configuration fields, showing default values redpanda: seed_brokers: [] # No default (optional) client_id: benthos tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) metadata_max_age: 5m request_timeout_overhead: 10s conn_idle_timeout: 20s pipeline_id: "" logs_topic: "" logs_level: info status_topic: "" partitioner: "" # No default (optional) idempotent_write: true compression: "" # No default (optional) timeout: 10s max_message_bytes: 1MB broker_write_max_bytes: 100MB allow_auto_topic_creation: true ``` ## [](#pipeline-example)Pipeline example This data pipeline reads data from `topic_A` and `topic_B` on a Redpanda cluster, and then writes the data to `topic_C` on the same cluster. The cluster details are configured within the `redpanda` configuration block, so you only need to configure them once. This is a useful feature when you have multiple inputs and outputs in the same data pipeline that need to connect to the same cluster. ```none input: redpanda_common: topics: [ topic_A, topic_B ] output: redpanda_common: topic: topic_C key: ${! @id } redpanda: seed_brokers: [ "127.0.0.1:9092" ] tls: enabled: true sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ## [](#fields)Fields ### [](#seed_brokers)`seed_brokers` A list of broker addresses to connect to in order. Use commas to separate multiple addresses in a single list item. **Type**: `array` ```yml # Examples seed_brokers: - localhost:9092 seed_brokers: - foo:9092 - bar:9092 seed_brokers: - foo:9092,bar:9092 ``` ### [](#client_id)`client_id` An identifier for the client connection. **Type**: `string` **Default**: `benthos` ### [](#tls)`tls` Override system defaults with custom TLS settings. **Type**: `object` ### [](#tls-enabled)`tls.enabled` Whether custom TLS settings are enabled. **Type**: `bool` **Default**: `false` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. **Type**: `bool` **Default**: `false` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. **Type**: `bool` **Default**: `false` Requires version 3.45.0 or newer ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent trusted root certificate, through possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yml # Example root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent trusted root certificate, through possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yml # Example root_cas_file: ./root_cas.pem ``` ### [](#tls-client_certs)`tls.client_certs` A list of client certificates to use. For each certificate, specify either the fields `cert` and `key` or `cert_file` and `key_file`. **Type**: `array` **Default**: `[]` ```yml # Examples client_certs: - cert: foo key: bar client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` The plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` The plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` The plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. > ⚠️ **WARNING** > > The `pbeWithMD5AndDES-CBC` algorithm does not authenticate ciphertext, and is vulnerable to padding oracle attacks which may allow an attacker to recover the plain text password. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yml # Examples password: foo password: ${KEY_PASSWORD} ``` ### [](#sasl)`sasl` Specify one or more methods or mechanisms of SASL authentication. They are tried in order. If the broker supports the first SASL mechanism, all connections use it. If the first mechanism fails, the client picks the first supported mechanism. If the broker does not support any client mechanisms, all connections fail. **Type**: `array` ```yml # Example sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ### [](#sasl-mechanism)`sasl[].mechanism` The SASL mechanism to use. **Type**: `string` | Option | Summary | | --- | --- | | AWS_MSK_IAM | AWS IAM-based authentication as specified by the aws-msk-iam-auth Java library. | | OAUTHBEARER | OAuth Bearer-based authentication. | | PLAIN | Plain text authentication. | | SCRAM-SHA-256 | SCRAM-based authentication as specified in RFC5802. | | SCRAM-SHA-512 | SCRAM-based authentication as specified in RFC5802. | | none | Disable SASL authentication | ### [](#sasl-username)`sasl[].username` A username for `PLAIN` or `SCRAM-*` authentication. **Type**: `string` **Default**: `""` ### [](#sasl-password)`sasl[].password` A password for `PLAIN` or `SCRAM-*` authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#sasl-token)`sasl[].token` The token to use for a single session’s `OAUTHBEARER` authentication. **Type**: `string` **Default**: `""` ### [](#sasl-extensions)`sasl[].extensions` Key/value pairs to add to `OAUTHBEARER` authentication requests. **Type**: `object` ### [](#sasl-aws)`sasl[].aws` AWS specific fields for when the `mechanism` is set to `AWS_MSK_IAM`. **Type**: `object` ### [](#sasl-aws-region)`sasl[].aws.region` The AWS region to target. **Type**: `string` **Default**: `""` ### [](#sasl-aws-endpoint)`sasl[].aws.endpoint` Specify a custom endpoint for the AWS API. **Type**: `string` **Default**: `""` ### [](#sasl-aws-credentials)`sasl[].aws.credentials` Manually configure the AWS credentials to use (optional). For more information, see the [Amazon Web Services guide](../../../guides/cloud/aws/). **Type**: `object` ### [](#sasl-aws-credentials-profile)`sasl[].aws.credentials.profile` The profile from `~/.aws/credentials` to use. **Type**: `string` **Default**: `""` ### [](#sasl-aws-credentials-id)`sasl[].aws.credentials.id` The ID of the AWS credentials to use. **Type**: `string` **Default**: `""` ### [](#sasl-aws-credentials-secret)`sasl[].aws.credentials.secret` The secret for the AWS credentials in use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#sasl-aws-credentials-token)`sasl[].aws.credentials.token` The token for the AWS credentials in use. This is a required value for short-term credentials. **Type**: `string` **Default**: `""` ### [](#sasl-aws-credentials-from_ec2_role)`sasl[].aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume an [IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). **Type**: `bool` **Default**: `false` Requires version 4.2.0 or newer ### [](#sasl-aws-credentials-role)`sasl[].aws.credentials.role` The role ARN to assume. **Type**: `string` **Default**: `""` ### [](#sasl-aws-credentials-role_external_id)`sasl[].aws.credentials.role_external_id` An external ID to use when assuming a role. **Type**: `string` **Default**: `""` ### [](#metadata_max_age)`metadata_max_age` The maximum period of time after which metadata is refreshed. **Type**: `string` **Default**: `5m` ### [](#request_timeout_overhead)`request_timeout_overhead` Grants an additional buffer or overhead to requests that have timeout fields defined. This field is based on the behavior of Apache Kafka’s `request.timeout.ms` parameter, but with the option to extend the timeout deadline. **Type**: `string` **Default**: `10s` ### [](#conn_idle_timeout)`conn_idle_timeout` Define how long connections can remain idle before they are closed. **Type**: `string` ### [](#pipeline_id)`pipeline_id` The ID of a Redpanda Connect data pipeline (optional). When specified, the pipeline ID is written to all logs and status updates sent to the configured topics. **Type**: `string` **Default**: `""` ### [](#logs_topic)`logs_topic` The topic that logs are sent to. **Type**: `string` **Default**: `""` ```yml # Example logs_topic: __redpanda.connect.logs ``` ### [](#logs_level)`logs_level` The logging level of logs sent to Redpanda. **Type**: `string` **Default**: `info` **Options**: `debug`, `info`, `warn`, `error` ### [](#status_topic)`status_topic` The topic that status updates are sent to. For full details of the schema for status updates, see the [object specification](https://github.com/redpanda-data/connect/blob/main/internal/protoconnect/status.pb.go). **Type**: `string` **Default**: `""` ```yml # Example status_topic: __redpanda.connect.status ``` ### [](#partitioner)`partitioner` Override the default murmur2 hashing partitioner. **Type**: `string` | Option | Summary | | --- | --- | | least_backup | Chooses the least backed up partition. The partition with the fewest buffered records. Partitions are selected per batch. | | manual | Manually select a partition for each message. You must also specify a value for the partition field. | | murmur2_hash | Kafka’s default hash algorithm that uses a 32-bit murmur2 hash of the key to compute the partition for the record. | | round_robin | Does a round robin of messages through all available partitions. This algorithm has lower throughput and causes higher CPU load on brokers, but is useful if you want to ensure an even distribution of records to partitions. | ### [](#idempotent_write)`idempotent_write` Enable the idempotent write producer option. This requires the `IDEMPOTENT_WRITE` permission on `CLUSTER`. Disable this option if the `IDEMPOTENT_WRITE` permission is not available. **Type**: `bool` **Default**: `true` ### [](#compression)`compression` Set an explicit compression type (optional). The default preference is to use `snappy` when the broker supports it. Otherwise, use `none`. **Type**: `string` Options: `lz4` , `snappy` , `gzip` , `none` , `zstd` ### [](#timeout)`timeout` The maximum period of time allowed for sending log or status update messages before a request is abandoned and a retry attempted. **Type**: `string` **Default**: `10s` ### [](#max_message_bytes)`max_message_bytes` The maximum size of an individual message in bytes. Messages larger than this value are rejected. This field is equivalent to Kafka’s `max.message.bytes`. **Type**: `string` **Default**: `1MB` ```yml # Examples max_message_bytes: 100MB max_message_bytes: 50mib ``` ### [](#broker_write_max_bytes)`broker_write_max_bytes` The upper bound for the number of bytes written to a broker connection in a single write. This field corresponds to Kafka’s `socket.request.max.bytes`. **Type**: `string` **Default**: `"100MB"` ```yml # Examples broker_write_max_bytes: 128MB broker_write_max_bytes: 50mib ``` --- # Page 309: Scanners **URL**: https://docs.redpanda.com/redpanda-connect/components/scanners/about.md --- # Scanners --- title: Scanners latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: scanners/about page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: scanners/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/scanners/about.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-09" --- For such inputs it’s necessary to define a mechanism by which the stream of source bytes can be chopped into smaller logical messages, processed and outputted as a continuous process whilst the stream is being read, as this dramatically reduces the memory usage of Redpanda Connect as a whole and results in a more fluid flow of data. The way in which we define this chopping mechanism is through scanners, configured as a field on each input that requires one. For example, if we wished to consume files line-by-line, which each individual line being processed as a discrete message, we could use the [`lines` scanner](../lines/) with our [`file` input](../../inputs/file/): ## Common ```yaml input: file: paths: [ "./*.txt" ] scanner: lines: {} ``` ## Advanced ```yaml # Instead of newlines, use a custom delimiter: input: file: paths: [ "./*.txt" ] scanner: lines: custom_delimiter: "---END---" max_buffer_size: 100_000_000 # 100MB line buffer ``` A scanner is a plugin similar to any other core Redpanda Connect component (inputs, processors, outputs, etc), which means it’s possible to define your own scanners that can be utilized by inputs that need them. --- # Page 310: avro **URL**: https://docs.redpanda.com/redpanda-connect/components/scanners/avro.md --- # avro --- title: avro latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: scanners/avro page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: scanners/avro.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/scanners/avro.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Scanner ▼ [Scanner](/redpanda-connect/components/scanners/avro/)[Processor](/redpanda-connect/components/processors/avro/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/scanners/avro/ "View the Cloud version of this component"), Self-Managed Consume a stream of Avro OCF datum. #### Common ```yml scanners: avro: ``` #### Advanced ```yml scanners: avro: raw_json: false ``` ## [](#avro-json-format)Avro JSON format This scanner creates documents formatted as [Avro JSON](https://avro.apache.org/docs/current/specification/) when decoding with Avro schemas. In this format, the value of a union is encoded in JSON as follows: - If the union’s type is `null`, it is encoded as a JSON `null`. - Otherwise, the union is encoded as a JSON object with one name/value pair. The `"name"` is the type’s name and the `"value"` is the recursively encoded value. For Avro’s named types (record, fixed or enum), the user-specified name is used. For other types, the type name is used. For example, the union schema `["null","string","Transaction"]`, where `Transaction` is a record name, would encode: - The `null` as a JSON `null` - The string `"a"` as `{"string": "a"}` - A `Transaction` instance as `{"Transaction": {…​}}`, where `{…​}` indicates the JSON encoding of a `Transaction` instance Alternatively, you can create documents in [standard/raw JSON format](https://pkg.go.dev/github.com/linkedin/goavro/v2#NewCodecForStandardJSONFull) by setting the field [`raw_json`](#raw_json) to `true`. ## [](#metadata)Metadata This scanner emits the following metadata for each message: - The `@avro_schema` field: The canonical Avro schema. - The `@avro_schema_fingerprint` field: The schema ID or fingerprint. ## [](#fields)Fields ### [](#raw_json)`raw_json` Whether to decode messages into normal JSON rather than [Avro JSON](https://avro.apache.org/docs/current/specification/_print/#json-encoding). When true, this unwraps union values (bare values instead of {"type": value} wrappers). **Type**: `bool` **Default**: `false` --- # Page 311: chunker **URL**: https://docs.redpanda.com/redpanda-connect/components/scanners/chunker.md --- # chunker --- title: chunker latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: scanners/chunker page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: scanners/chunker.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/scanners/chunker.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/scanners/chunker/ "View the Cloud version of this component"), Self-Managed Split an input stream into chunks of a given number of bytes. ```yml # Config fields, showing default values chunker: size: 0 # No default (required) ``` ## [](#fields)Fields ### [](#size)`size` The size of each chunk in bytes. **Type**: `int` --- # Page 312: csv **URL**: https://docs.redpanda.com/redpanda-connect/components/scanners/csv.md --- # csv --- title: csv latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: scanners/csv page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: scanners/csv.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/scanners/csv.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-01-05" --- **Type:** Scanner ▼ [Scanner](/redpanda-connect/components/scanners/csv/)[Input](/redpanda-connect/components/inputs/csv/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/scanners/csv/ "View the Cloud version of this component"), Self-Managed Consume comma-separated values row by row, including support for custom delimiters. ```yml # Config fields, showing default values csv: custom_delimiter: "" # No default (optional) parse_header_row: true lazy_quotes: false continue_on_error: false ``` ## [](#metadata)Metadata This scanner adds the following metadata to each message: - `csv_row` The index of each row, beginning at 0. ## [](#fields)Fields ### [](#continue_on_error)`continue_on_error` If a row fails to parse due to any error emit an empty message marked with the error and then continue consuming subsequent rows when possible. This can sometimes be useful in situations where input data contains individual rows which are malformed. However, when a row encounters a parsing error it is impossible to guarantee that following rows are valid, as this indicates that the input data is unreliable and could potentially emit misaligned rows. **Type**: `bool` **Default**: `false` ### [](#custom_delimiter)`custom_delimiter` Use a provided custom delimiter instead of the default comma. **Type**: `string` ### [](#lazy_quotes)`lazy_quotes` If set to `true`, a quote may appear in an unquoted field and a non-doubled quote may appear in a quoted field. **Type**: `bool` **Default**: `false` ### [](#parse_header_row)`parse_header_row` Whether to reference the first row as a header row. If set to true the output structure for messages will be an object where field keys are determined by the header row. Otherwise, each message will consist of an array of values from the corresponding CSV row. **Type**: `bool` **Default**: `true` --- # Page 313: decompress **URL**: https://docs.redpanda.com/redpanda-connect/components/scanners/decompress.md --- # decompress --- title: decompress latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: scanners/decompress page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: scanners/decompress.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/scanners/decompress.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Type:** Scanner ▼ [Scanner](/redpanda-connect/components/scanners/decompress/)[Processor](/redpanda-connect/components/processors/decompress/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/scanners/decompress/ "View the Cloud version of this component"), Self-Managed Decompress the stream of bytes according to an algorithm, before feeding it into a child scanner. ```yml # Config fields, showing default values decompress: algorithm: "" # No default (required) into: to_the_end: {} ``` ## [](#fields)Fields ### [](#algorithm)`algorithm` One of `gzip`, `pgzip`, `zlib`, `bzip2`, `flate`, `snappy`, `lz4`, `zstd`. **Type**: `string` ### [](#into)`into` The child scanner to feed the decompressed stream into. **Type**: `scanner` **Default**: ```yaml to_the_end: {} ``` --- # Page 314: json_array **URL**: https://docs.redpanda.com/redpanda-connect/components/scanners/json_array.md --- # json\_array --- title: json_array latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: scanners/json_array page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: scanners/json_array.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/scanners/json_array.adoc description: Consumes a stream of one or more JSON elements within a top level array. page-git-created-date: "2025-09-25" page-git-modified-date: "2025-09-26" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/scanners/json_array/ "View the Cloud version of this component"), Self-Managed **Introduced in version 4.65.0.** Consumes a stream of one or more JSON elements within a top level array. This scanner is useful for: - Processing exports from systems that generate a JSON array as the top-level JSON structure (for example, logs, bulk exports, etc). - Efficiently breaking up large files with many objects into individual events/messages. Suppose you have a file `events.json`: `events.json` ```json [ {"event": "login", "user": "alice"}, {"event": "logout", "user": "bob"}, {"event": "purchase", "user": "carol", "amount": 42} ] ``` The configuration to process this file is: ```yaml input: file: paths: [ "./events.json" ] scanner: json_array: {} ``` Result: Each event in the array is processed as a separate message. ## [](#requirements)Requirements The `json_array` scanner expects the input to be a single JSON array, where each array element is a JSON object or value. ## [](#fields)Fields The `json_array` scanner has no required fields. You declare it as `{}` in your config. ```yaml json_array: {} ``` --- # Page 315: json_documents **URL**: https://docs.redpanda.com/redpanda-connect/components/scanners/json_documents.md --- # json\_documents --- title: json_documents latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: scanners/json_documents page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: scanners/json_documents.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/scanners/json_documents.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/scanners/json_documents/ "View the Cloud version of this component"), Self-Managed Consumes a stream of one or more JSON documents. Introduced in version 4.27.0. ```yml # Config fields, showing default values json_documents: {} ``` --- # Page 316: lines **URL**: https://docs.redpanda.com/redpanda-connect/components/scanners/lines.md --- # lines --- title: lines latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: scanners/lines page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: scanners/lines.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/scanners/lines.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/scanners/lines/ "View the Cloud version of this component"), Self-Managed Split an input stream into a message per line of data. ```yml # Config fields, showing default values lines: custom_delimiter: "" # No default (optional) max_buffer_size: 65536 omit_empty: false ``` ## [](#fields)Fields ### [](#custom_delimiter)`custom_delimiter` Use a provided custom delimiter for detecting the end of a line rather than a single line break. **Type**: `string` ### [](#max_buffer_size)`max_buffer_size` Set the maximum buffer size for storing line data, this limits the maximum size that a line can be without causing an error. **Type**: `int` **Default**: `65536` ### [](#omit_empty)`omit_empty` Omit empty lines. **Type**: `bool` **Default**: `false` --- # Page 317: re_match **URL**: https://docs.redpanda.com/redpanda-connect/components/scanners/re_match.md --- # re\_match --- title: re_match latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: scanners/re_match page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: scanners/re_match.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/scanners/re_match.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/scanners/re_match/ "View the Cloud version of this component"), Self-Managed Split an input stream into segments matching against a regular expression. ```yml # Config fields, showing default values re_match: pattern: (?m)^\d\d:\d\d:\d\d # No default (required) max_buffer_size: 65536 ``` ## [](#fields)Fields ### [](#max_buffer_size)`max_buffer_size` Set the maximum buffer size for storing line data, this limits the maximum size that a message can be without causing an error. **Type**: `int` **Default**: `65536` ### [](#pattern)`pattern` The pattern to match against. **Type**: `string` ```yaml # Examples: pattern: (?m)^\d\d:\d\d:\d\d ``` --- # Page 318: skip_bom **URL**: https://docs.redpanda.com/redpanda-connect/components/scanners/skip_bom.md --- # skip\_bom --- title: skip_bom latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: scanners/skip_bom page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: scanners/skip_bom.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/scanners/skip_bom.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-06-25" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/scanners/skip_bom/ "View the Cloud version of this component"), Self-Managed Skip one or more byte order marks for each opened child scanner. ```yml # Config fields, showing default values skip_bom: into: to_the_end: {} ``` ## [](#fields)Fields ### [](#into)`into` The child scanner to feed the resulting stream into. **Type**: `scanner` **Default**: ```yaml to_the_end: {} ``` --- # Page 319: switch **URL**: https://docs.redpanda.com/redpanda-connect/components/scanners/switch.md --- # switch --- title: switch latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: scanners/switch page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: scanners/switch.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/scanners/switch.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-20" --- **Type:** Scanner ▼ [Scanner](/redpanda-connect/components/scanners/switch/)[Output](/redpanda-connect/components/outputs/switch/)[Processor](/redpanda-connect/components/processors/switch/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/scanners/switch/ "View the Cloud version of this component"), Self-Managed Select a child scanner dynamically for source data based on factors such as the filename. ```yml # Config fields, showing default values switch: [] # No default (required) ``` This scanner outlines a list of potential child scanner candidates to be chosen, and for each source of data the first candidate to pass will be selected. A candidate without any conditions acts as a catch-all and will pass for every source, it is recommended to always have a catch-all scanner at the end of your list. If a given source of data does not pass a candidate an error is returned and the data is rejected. ## [](#fields)Fields ### [](#re_match_name)`re_match_name` A regular expression to test against the name of each source of data fed into the scanner (filename or equivalent). If this pattern matches the child scanner is selected. **Type**: `string` ### [](#scanner)`scanner` The scanner to activate if this candidate passes. **Type**: `scanner` ## [](#examples)Examples ### [](#switch-based-on-file-name)Switch based on file name In this example a file input chooses a scanner based on the extension of each file ```yaml input: file: paths: [ ./data/* ] scanner: switch: - re_match_name: '\.avro$' scanner: { avro: {} } - re_match_name: '\.csv$' scanner: { csv: {} } - re_match_name: '\.csv.gz$' scanner: decompress: algorithm: gzip into: csv: {} - re_match_name: '\.tar$' scanner: { tar: {} } - re_match_name: '\.tar.gz$' scanner: decompress: algorithm: gzip into: tar: {} - scanner: { to_the_end: {} } ``` --- # Page 320: tar **URL**: https://docs.redpanda.com/redpanda-connect/components/scanners/tar.md --- # tar --- title: tar latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: scanners/tar page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: scanners/tar.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/scanners/tar.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/scanners/tar/ "View the Cloud version of this component"), Self-Managed Consume a tar archive file by file. ```yml # Config fields, showing default values tar: {} ``` ## [](#metadata)Metadata This scanner adds the following metadata to each message: - `tar_name` --- # Page 321: to_the_end **URL**: https://docs.redpanda.com/redpanda-connect/components/scanners/to_the_end.md --- # to\_the\_end --- title: to_the_end latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: scanners/to_the_end page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: scanners/to_the_end.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/scanners/to_the_end.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/scanners/to_the_end/ "View the Cloud version of this component"), Self-Managed Read the input stream all the way until the end and deliver it as a single message. ```yml # Config fields, showing default values to_the_end: {} ``` > ⚠️ **CAUTION** > > Some sources of data may not have a logical end, therefore caution should be made to exclusively use this scanner when the end of an input stream is clearly defined (and well within memory). --- # Page 322: Tracers **URL**: https://docs.redpanda.com/redpanda-connect/components/tracers/about.md --- # Tracers --- title: Tracers latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: tracers/about page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: tracers/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/tracers/about.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- A tracer type represents a destination for Redpanda Connect to send tracing events to such as [Jaeger](https://www.jaegertracing.io/). When a tracer is configured all messages will be allocated a root span during ingestion that represents their journey through a Redpanda Connect pipeline. Many Redpanda Connect processors create spans, and so tracing is a great way to analyse the pathways of individual messages as they progress through a Redpanda Connect instance. Some inputs, such as `http_server` and `http_client`, are capable of extracting a root span from the source of the message (HTTP headers). This is a work in progress and should eventually expand so that all inputs have a way of doing so. Other inputs, such as `kafka` can be configured to extract a root span by using the `extract_tracing_map` field. A tracer config section looks like this: ```yaml tracer: jaeger: agent_address: localhost:6831 sampler_type: const sampler_param: 1 ``` > ⚠️ **CAUTION** > > Although the configuration spec of this component is stable the format of spans, tags and logs created by Redpanda Connect is subject to change as it is tuned for improvement. --- # Page 323: gcp_cloudtrace **URL**: https://docs.redpanda.com/redpanda-connect/components/tracers/gcp_cloudtrace.md --- # gcp\_cloudtrace --- title: gcp_cloudtrace latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: tracers/gcp_cloudtrace page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: tracers/gcp_cloudtrace.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/tracers/gcp_cloudtrace.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/tracers/gcp_cloudtrace/ "View the Cloud version of this component"), Self-Managed Send tracing events to a [Google Cloud Trace](https://cloud.google.com/trace). Introduced in version 4.2.0. #### Common ```yml tracers: gcp_cloudtrace: project: "" # No default (required) sampling_ratio: 1 flush_interval: "" # No default (optional) ``` #### Advanced ```yml tracers: gcp_cloudtrace: project: "" # No default (required) sampling_ratio: 1 tags: {} flush_interval: "" # No default (optional) ``` ## [](#fields)Fields ### [](#flush_interval)`flush_interval` The period of time between each flush of tracing spans. **Type**: `string` ### [](#project)`project` The google project with Cloud Trace API enabled. If this is omitted then the Google Cloud SDK will attempt auto-detect it from the environment. **Type**: `string` ### [](#sampling_ratio)`sampling_ratio` Sets the ratio of traces to sample. Tuning the sampling ratio is recommended for high-volume production workloads. **Type**: `float` **Default**: `1` ```yaml # Examples: sampling_ratio: 1 ``` ### [](#tags)`tags` A map of tags to add to tracing spans. **Type**: `string` **Default**: `{}` --- # Page 324: jaeger **URL**: https://docs.redpanda.com/redpanda-connect/components/tracers/jaeger.md --- # jaeger --- title: jaeger latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: tracers/jaeger page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: tracers/jaeger.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/tracers/jaeger.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Available in:** Self-Managed Send tracing events to a [Jaeger](https://www.jaegertracing.io/) agent or collector. #### Common ```yml tracers: jaeger: agent_address: "" collector_url: "" sampler_type: const flush_interval: "" # No default (optional) ``` #### Advanced ```yml tracers: jaeger: agent_address: "" collector_url: "" sampler_type: const sampler_param: 1 tags: {} flush_interval: "" # No default (optional) ``` ## [](#fields)Fields ### [](#agent_address)`agent_address` The address of a Jaeger agent to send tracing events to. **Type**: `string` **Default**: `""` ```yaml # Examples: agent_address: jaeger-agent:6831 ``` ### [](#collector_url)`collector_url` The URL of a Jaeger collector to send tracing events to. If set, this will override `agent_address`. Requires version 3.38.0 or later. **Type**: `string` **Default**: `""` ```yaml # Examples: collector_url: https://jaeger-collector:14268/api/traces ``` ### [](#flush_interval)`flush_interval` The period of time between each flush of tracing spans. **Type**: `string` ### [](#sampler_param)`sampler_param` A parameter to use for sampling. This field is unused for some sampling types. **Type**: `float` **Default**: `1` ### [](#sampler_type)`sampler_type` The sampler type to use. **Type**: `string` **Default**: `const` | Option | Summary | | --- | --- | | const | Sample a percentage of traces. 1 or more means all traces are sampled, 0 means no traces are sampled and anything in between means a percentage of traces are sampled. Tuning the sampling rate is recommended for high-volume production workloads. | ### [](#tags)`tags` A map of tags to add to tracing spans. **Type**: `string` **Default**: `{}` --- # Page 325: none **URL**: https://docs.redpanda.com/redpanda-connect/components/tracers/none.md --- # none --- title: none latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: tracers/none page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: tracers/none.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/tracers/none.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- **Type:** Tracer ▼ [Tracer](/redpanda-connect/components/tracers/none/)[Buffer](/redpanda-connect/components/buffers/none/)[Metric](/redpanda-connect/components/metrics/none/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/tracers/none/ "View the Cloud version of this component"), Self-Managed Do not send tracing events anywhere. ```yml # Config fields, showing default values tracer: none: {} ``` --- # Page 326: open_telemetry_collector **URL**: https://docs.redpanda.com/redpanda-connect/components/tracers/open_telemetry_collector.md --- # open\_telemetry\_collector --- title: open_telemetry_collector latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: tracers/open_telemetry_collector page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: tracers/open_telemetry_collector.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/tracers/open_telemetry_collector.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-06" --- **Type:** Tracer ▼ [Tracer](/redpanda-connect/components/tracers/open_telemetry_collector/)Metric **Available in:** Self-Managed Send tracing events to an [Open Telemetry collector](https://opentelemetry.io/docs/collector/). #### Common ```yml tracers: open_telemetry_collector: service: benthos http: [] # No default (required) grpc: [] # No default (required) sampling: enabled: false ratio: "" # No default (optional) ``` #### Advanced ```yml tracers: open_telemetry_collector: service: benthos http: [] # No default (required) grpc: [] # No default (required) tags: {} sampling: enabled: false ratio: "" # No default (optional) ``` ## [](#fields)Fields ### [](#grpc)`grpc[]` A list of grpc collectors. **Type**: `object` ### [](#grpc-address)`grpc[].address` The endpoint of a collector to send tracing events to. **Type**: `string` ```yaml # Examples: address: localhost:4317 ``` ### [](#grpc-secure)`grpc[].secure` Connect to the collector with client transport security **Type**: `bool` **Default**: `false` ### [](#http)`http[]` A list of http collectors. **Type**: `object` ### [](#http-address)`http[].address` The endpoint of a collector to send tracing events to. **Type**: `string` ```yaml # Examples: address: localhost:4318 ``` ### [](#http-secure)`http[].secure` Connect to the collector over HTTPS **Type**: `bool` **Default**: `false` ### [](#sampling)`sampling` Settings for trace sampling. Sampling is recommended for high-volume production workloads. Requires version 4.25.0 or later. **Type**: `object` ### [](#sampling-enabled)`sampling.enabled` Whether to enable sampling. **Type**: `bool` **Default**: `false` ### [](#sampling-ratio)`sampling.ratio` Sets the ratio of traces to sample. **Type**: `float` ```yaml # Examples: ratio: 0.85 # --- ratio: 0.5 ``` ### [](#service)`service` The name of the service in traces. **Type**: `string` **Default**: `benthos` ### [](#tags)`tags` A map of tags to add to all tracing spans. **Type**: `string` **Default**: `{}` --- # Page 327: redpanda **URL**: https://docs.redpanda.com/redpanda-connect/components/tracers/redpanda.md --- # redpanda --- title: redpanda latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: tracers/redpanda page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: tracers/redpanda.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/components/pages/tracers/redpanda.adoc description: Send tracing events to a Redpanda topic. page-git-created-date: "2025-12-02" page-git-modified-date: "2025-12-03" --- **Type:** Tracer ▼ [Tracer](/redpanda-connect/components/tracers/redpanda/)[Cache](/redpanda-connect/components/caches/redpanda/)[Input](/redpanda-connect/components/inputs/redpanda/)[Output](/redpanda-connect/components/outputs/redpanda/) **Available in:** [Cloud](/redpanda-cloud/develop/connect/components/tracers/redpanda/ "View the Cloud version of this component"), Self-Managed Export distributed tracing data to a Redpanda topic, enabling you to monitor and debug your Redpanda Connect pipelines. Traces are exported in OpenTelemetry format as JSON, allowing integration with observability platforms like Jaeger, Grafana Tempo, or custom trace consumers. #### Common ```yml tracers: redpanda: seed_brokers: [] # No default (required) topic: otel-traces format: json schema_registry: url: "" # No default (optional) tls: skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth2: enabled: false client_key: "" client_secret: "" token_url: "" scopes: [] endpoint_params: {} oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} service: redpanda-connect sampling: enabled: false ratio: "" # No default (optional) ``` #### Advanced ```yml tracers: redpanda: seed_brokers: [] # No default (required) client_id: redpanda-connect tls: enabled: false skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] sasl: [] # No default (optional) metadata_max_age: 1m request_timeout_overhead: 10s conn_idle_timeout: 20s tcp: connect_timeout: 0s keep_alive: idle: 15s interval: 15s count: 9 tcp_user_timeout: 0s partitioner: "" # No default (optional) idempotent_write: true compression: "" # No default (optional) allow_auto_topic_creation: true timeout: 10s max_message_bytes: 1MiB broker_write_max_bytes: 100MiB topic: otel-traces format: json schema_registry: url: "" # No default (optional) tls: skip_cert_verify: false enable_renegotiation: false root_cas: "" root_cas_file: "" client_certs: [] oauth2: enabled: false client_key: "" client_secret: "" token_url: "" scopes: [] endpoint_params: {} oauth: enabled: false consumer_key: "" consumer_secret: "" access_token: "" access_token_secret: "" basic_auth: enabled: false username: "" password: "" jwt: enabled: false private_key_file: "" signing_method: "" claims: {} headers: {} service: redpanda-connect tags: {} sampling: enabled: false ratio: "" # No default (optional) ``` This tracer automatically captures trace spans as messages flow through your pipeline, recording timing information, component metadata, and error details. Use this to: - **Track message flow** through complex pipelines with multiple processors. - **Identify performance bottlenecks** by analyzing span durations. - **Debug failures** by examining trace context and error details. - **Monitor pipeline health** across distributed Redpanda Connect instances. - **Correlate activity** across multiple services using trace IDs. The tracer writes to a dedicated Redpanda topic that can be consumed by trace analysis tools. Configure sampling to control trace volume in high-throughput environments. ## [](#fields)Fields ### [](#allow_auto_topic_creation)`allow_auto_topic_creation` Whether to automatically create the trace topic if it doesn’t exist. If false, the topic must be created manually before starting the tracer. **Type**: `bool` **Default**: `true` ### [](#broker_write_max_bytes)`broker_write_max_bytes` The maximum number of bytes this output can write to a broker connection in a single write. This field corresponds to Kafka’s `socket.request.max.bytes`. **Type**: `string` **Default**: `100MiB` ```yaml # Examples: broker_write_max_bytes: 128MB # --- broker_write_max_bytes: 50mib ``` ### [](#client_id)`client_id` An identifier for the client connection. This appears in broker logs and metrics to help identify which Redpanda Connect instance is sending traces. **Type**: `string` **Default**: `redpanda-connect` ### [](#compression)`compression` Compression codec to use for trace messages. Options include `gzip`, `snappy`, `lz4`, `zstd`, or none. Compression can reduce network bandwidth and storage costs. **Type**: `string` **Options**: `lz4`, `snappy`, `gzip`, `none`, `zstd` ### [](#conn_idle_timeout)`conn_idle_timeout` The maximum duration that connections can remain idle before they are automatically closed. This field accepts Go duration format strings such as `100ms`, `1s`, or `5s`. **Type**: `string` **Default**: `20s` ### [](#format)`format` The format for trace data. Currently only `json` is supported, which exports OpenTelemetry spans as JSON messages. **Type**: `string` **Default**: `json` | Option | Summary | | --- | --- | | json | Emit in JSON Format | | protobuf | Emit in Protobuf Format | | schema-registry-json | Emit in JSON Format with Schema Registry encoding | | schema-registry-protobuf | Emit in Protobuf Format with Schema Registry encoding | ### [](#idempotent_write)`idempotent_write` Enable idempotent writes to prevent duplicate trace messages in case of retries. Recommended for production environments. **Type**: `bool` **Default**: `true` ### [](#max_message_bytes)`max_message_bytes` The maximum size of individual trace messages. Traces exceeding this size will be truncated or dropped. **Type**: `string` **Default**: `1MiB` ```yaml # Examples: max_message_bytes: 100MB # --- max_message_bytes: 50mib ``` ### [](#metadata_max_age)`metadata_max_age` The maximum age of cached cluster metadata before it is refreshed. Reducing this value can help detect cluster changes faster but increases metadata requests. **Type**: `string` **Default**: `1m` ### [](#partitioner)`partitioner` Override the default partitioner for trace messages. By default, traces are distributed across partitions for load balancing. **Type**: `string` | Option | Summary | | --- | --- | | least_backup | Chooses the least backed up partition (the partition with the fewest amount of buffered records). Partitions are selected per batch. | | manual | Manually select a partition for each message, requires the field partition to be specified. | | murmur2_hash | Kafka’s default hash algorithm that uses a 32-bit murmur2 hash of the key to compute which partition the record will be on. | | round_robin | Round-robin’s messages through all available partitions. This algorithm has lower throughput and causes higher CPU load on brokers, but can be useful if you want to ensure an even distribution of records to partitions. | ### [](#request_timeout_overhead)`request_timeout_overhead` Additional time to apply as overhead when calculating request deadlines. This buffer helps prevent premature timeouts. **Type**: `string` **Default**: `10s` ### [](#sampling)`sampling` Configure trace sampling to control the volume of trace data. Sampling is essential for high-throughput pipelines to prevent trace data from overwhelming your observability infrastructure. **Type**: `object` ### [](#sampling-enabled)`sampling.enabled` Whether to enable trace sampling. When disabled, all traces are exported. When enabled, traces are sampled according to the configured ratio. **Type**: `bool` **Default**: `false` ### [](#sampling-ratio)`sampling.ratio` The sampling ratio as a decimal between 0 and 1. For example, `0.1` samples 10% of traces, `0.01` samples 1%. Lower ratios reduce trace volume and overhead. For high-throughput production systems, start with 0.01-0.1 and adjust based on your needs. **Type**: `float` ```yaml # Examples: ratio: 0.05 # --- ratio: 0.85 # --- ratio: 0.5 ``` ### [](#sasl)`sasl[]` Specify one or more methods or mechanisms of SASL authentication, which are attempted in order. If the broker supports the first SASL mechanism, all connections use it. If the first mechanism fails, the client picks the first supported mechanism. If the broker does not support any client mechanisms, all connections fail. **Type**: `object` ```yaml # Examples: sasl: - mechanism: SCRAM-SHA-512 password: bar username: foo ``` ### [](#sasl-aws)`sasl[].aws` Contains AWS specific fields for when the `mechanism` is set to `AWS_MSK_IAM`. **Type**: `object` ### [](#sasl-aws-credentials)`sasl[].aws.credentials` Optional manual configuration of AWS credentials to use. More information can be found in [Amazon Web Services](../../../guides/cloud/aws/). **Type**: `object` ### [](#sasl-aws-credentials-from_ec2_role)`sasl[].aws.credentials.from_ec2_role` Use the credentials of a host EC2 machine configured to assume [an IAM role associated with the instance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). Requires version 4.2.0 or later. **Type**: `bool` ### [](#sasl-aws-credentials-id)`sasl[].aws.credentials.id` The ID of credentials to use. **Type**: `string` ### [](#sasl-aws-credentials-profile)`sasl[].aws.credentials.profile` A profile from `~/.aws/credentials` to use. **Type**: `string` ### [](#sasl-aws-credentials-role)`sasl[].aws.credentials.role` A role ARN to assume. **Type**: `string` ### [](#sasl-aws-credentials-role_external_id)`sasl[].aws.credentials.role_external_id` An external ID to provide when assuming a role. **Type**: `string` ### [](#sasl-aws-credentials-secret)`sasl[].aws.credentials.secret` The secret for the credentials being used. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` ### [](#sasl-aws-credentials-token)`sasl[].aws.credentials.token` The token for the credentials being used, required when using short term credentials. **Type**: `string` ### [](#sasl-aws-endpoint)`sasl[].aws.endpoint` Allows you to specify a custom endpoint for the AWS API. **Type**: `string` ### [](#sasl-aws-region)`sasl[].aws.region` The AWS region to target. **Type**: `string` ### [](#sasl-aws-tcp)`sasl[].aws.tcp` TCP socket configuration. **Type**: `object` ### [](#sasl-aws-tcp-connect_timeout)`sasl[].aws.tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-aws-tcp-keep_alive)`sasl[].aws.tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#sasl-aws-tcp-keep_alive-count)`sasl[].aws.tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#sasl-aws-tcp-keep_alive-idle)`sasl[].aws.tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-keep_alive-interval)`sasl[].aws.tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#sasl-aws-tcp-tcp_user_timeout)`sasl[].aws.tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#sasl-extensions)`sasl[].extensions` Key/value pairs to add to OAUTHBEARER authentication requests. **Type**: `string` ### [](#sasl-mechanism)`sasl[].mechanism` The SASL mechanism to use. **Type**: `string` | Option | Summary | | --- | --- | | AWS_MSK_IAM | AWS IAM based authentication as specified by the 'aws-msk-iam-auth' java library. | | OAUTHBEARER | OAuth Bearer based authentication. | | PLAIN | Plain text authentication. | | REDPANDA_CLOUD_SERVICE_ACCOUNT | Redpanda Cloud Service Account authentication when running in Redpanda Cloud. | | SCRAM-SHA-256 | SCRAM based authentication as specified in RFC5802. | | SCRAM-SHA-512 | SCRAM based authentication as specified in RFC5802. | | none | Disable sasl authentication | ### [](#sasl-password)`sasl[].password` A password to provide for PLAIN or SCRAM-\* authentication. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#sasl-token)`sasl[].token` The token to use for a single session’s OAUTHBEARER authentication. **Type**: `string` **Default**: `""` ### [](#sasl-username)`sasl[].username` A username to provide for PLAIN or SCRAM-\* authentication. **Type**: `string` **Default**: `""` ### [](#schema_registry)`schema_registry` Schema registry information to publish schemas for tracing data along with the data. **Type**: `object` ### [](#schema_registry-basic_auth)`schema_registry.basic_auth` Allows you to specify basic authentication. **Type**: `object` ### [](#schema_registry-basic_auth-enabled)`schema_registry.basic_auth.enabled` Whether to use basic authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-basic_auth-password)`schema_registry.basic_auth.password` A password to authenticate with. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-basic_auth-username)`schema_registry.basic_auth.username` A username to authenticate as. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt)`schema_registry.jwt` Beta Allows you to specify JWT authentication. **Type**: `object` ### [](#schema_registry-jwt-claims)`schema_registry.jwt.claims` A value used to identify the claims that issued the JWT. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-enabled)`schema_registry.jwt.enabled` Whether to use JWT authentication in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-jwt-headers)`schema_registry.jwt.headers` Add optional key/value headers to the JWT. **Type**: `object` **Default**: `{}` ### [](#schema_registry-jwt-private_key_file)`schema_registry.jwt.private_key_file` A file with the PEM encoded via PKCS1 or PKCS8 as private key. **Type**: `string` **Default**: `""` ### [](#schema_registry-jwt-signing_method)`schema_registry.jwt.signing_method` A method used to sign the token such as RS256, RS384, RS512 or EdDSA. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth)`schema_registry.oauth` Allows you to specify open authentication via OAuth version 1. **Type**: `object` ### [](#schema_registry-oauth-access_token)`schema_registry.oauth.access_token` A value used to gain access to the protected resources on behalf of the user. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-access_token_secret)`schema_registry.oauth.access_token_secret` A secret provided in order to establish ownership of a given access token. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_key)`schema_registry.oauth.consumer_key` A value used to identify the client to the service provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-consumer_secret)`schema_registry.oauth.consumer_secret` A secret used to establish ownership of the consumer key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth-enabled)`schema_registry.oauth.enabled` Whether to use OAuth version 1 in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-oauth2)`schema_registry.oauth2` Allows you to specify open authentication via OAuth version 2 using the client credentials token flow. **Type**: `object` ### [](#schema_registry-oauth2-client_key)`schema_registry.oauth2.client_key` A value used to identify the client to the token provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth2-client_secret)`schema_registry.oauth2.client_secret` A secret used to establish ownership of the client key. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-oauth2-enabled)`schema_registry.oauth2.enabled` Whether to use OAuth version 2 in requests. **Type**: `bool` **Default**: `false` ### [](#schema_registry-oauth2-endpoint_params)`schema_registry.oauth2.endpoint_params` A list of optional endpoint parameters, values should be arrays of strings. **Type**: `object` **Default**: `{}` ```yaml # Examples: endpoint_params: audience: - https://example.com resource: - https://api.example.com ``` ### [](#schema_registry-oauth2-scopes)`schema_registry.oauth2.scopes[]` A list of optional requested permissions. **Type**: `array` **Default**: `[]` ### [](#schema_registry-oauth2-token_url)`schema_registry.oauth2.token_url` The URL of the token provider. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls)`schema_registry.tls` Custom TLS settings can be used to override system defaults. **Type**: `object` ### [](#schema_registry-tls-client_certs)`schema_registry.tls.client_certs[]` A list of client certificates to use. For each certificate either the fields `cert` and `key`, or `cert_file` and `key_file` should be specified, but not both. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#schema_registry-tls-client_certs-cert)`schema_registry.tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-cert_file)`schema_registry.tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key)`schema_registry.tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-key_file)`schema_registry.tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#schema_registry-tls-client_certs-password)`schema_registry.tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#schema_registry-tls-enable_renegotiation)`schema_registry.tls.enable_renegotiation` Whether to allow the remote server to repeatedly request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#schema_registry-tls-root_cas)`schema_registry.tls.root_cas` An optional root certificate authority to use. This is a string, representing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#schema_registry-tls-root_cas_file)`schema_registry.tls.root_cas_file` An optional path of a root certificate authority file to use. This is a file, often with a .pem extension, containing a certificate chain from the parent trusted root certificate, to possible intermediate signing certificates, to the host certificate. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#schema_registry-tls-skip_cert_verify)`schema_registry.tls.skip_cert_verify` Whether to skip server side certificate verification. **Type**: `bool` **Default**: `false` ### [](#schema_registry-url)`schema_registry.url` The base URL of the schema registry service. **Type**: `string` ### [](#seed_brokers)`seed_brokers[]` A list of broker addresses to connect to in order. Use commas to separate multiple addresses in a single list item. **Type**: `array` ```yaml # Examples: seed_brokers: - "localhost:9092" # --- seed_brokers: - "foo:9092" - "bar:9092" # --- seed_brokers: - "foo:9092,bar:9092" ``` ### [](#service)`service` The service name to identify this Redpanda Connect instance in traces. This appears in trace visualizations and helps correlate traces across distributed systems. Use descriptive names like `order-processor` or `analytics-pipeline`. **Type**: `string` **Default**: `redpanda-connect` ### [](#tags)`tags` Custom key-value tags to attach to all traces from this instance. Use tags to add metadata like environment (`production`, `staging`), region, version, or instance identifiers. Tags appear as resource attributes in OpenTelemetry traces. **Type**: `string` **Default**: `{}` ### [](#tcp)`tcp` Configure TCP socket-level settings to optimize network performance and reliability. These low-level controls are useful for: - **High-latency networks**: Increase `connect_timeout` to allow more time for connection establishment - **Long-lived connections**: Configure `keep_alive` settings to detect and recover from stale connections - **Unstable networks**: Tune keep-alive probes to balance between quick failure detection and avoiding false positives - **Linux systems with specific requirements**: Use `tcp_user_timeout` (Linux 2.6.37+) to control data acknowledgment timeouts Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. **Type**: `object` ### [](#tcp-connect_timeout)`tcp.connect_timeout` Maximum amount of time a dial will wait for a connect to complete. Zero disables. **Type**: `string` **Default**: `0s` ### [](#tcp-keep_alive)`tcp.keep_alive` TCP keep-alive probe configuration. **Type**: `object` ### [](#tcp-keep_alive-count)`tcp.keep_alive.count` Maximum unanswered keep-alive probes before dropping the connection. Zero defaults to 9. **Type**: `int` **Default**: `9` ### [](#tcp-keep_alive-idle)`tcp.keep_alive.idle` Duration the connection must be idle before sending the first keep-alive probe. Zero defaults to 15s. Negative values disable keep-alive probes. **Type**: `string` **Default**: `15s` ### [](#tcp-keep_alive-interval)`tcp.keep_alive.interval` Duration between keep-alive probes. Zero defaults to 15s. **Type**: `string` **Default**: `15s` ### [](#tcp-tcp_user_timeout)`tcp.tcp_user_timeout` Maximum time to wait for acknowledgment of transmitted data before killing the connection. Linux-only (kernel 2.6.37+), ignored on other platforms. When enabled, keep\_alive.idle must be greater than this value per RFC 5482. Zero disables. **Type**: `string` **Default**: `0s` ### [](#timeout)`timeout` The maximum time to wait for trace messages to be acknowledged by the broker before considering the write failed. **Type**: `string` **Default**: `10s` ### [](#tls)`tls` Configure Transport Layer Security (TLS) settings to secure network connections. This includes options for standard TLS as well as mutual TLS (mTLS) authentication where both client and server authenticate each other using certificates. Key configuration options include `enabled` to enable TLS, `client_certs` for mTLS authentication, `root_cas`/`root_cas_file` for custom certificate authorities, and `skip_cert_verify` for development environments. **Type**: `object` ### [](#tls-client_certs)`tls.client_certs[]` A list of client certificates for mutual TLS (mTLS) authentication. Configure this field to enable mTLS, authenticating the client to the server with these certificates. You must set `tls.enabled: true` for the client certificates to take effect. **Certificate pairing rules**: For each certificate item, provide either: - Inline PEM data using both `cert` **and** `key` or - File paths using both `cert_file` **and** `key_file`. Mixing inline and file-based values within the same item is not supported. **Type**: `object` **Default**: `[]` ```yaml # Examples: client_certs: - cert: foo key: bar # --- client_certs: - cert_file: ./example.pem key_file: ./example.key ``` ### [](#tls-client_certs-cert)`tls.client_certs[].cert` A plain text certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-cert_file)`tls.client_certs[].cert_file` The path of a certificate to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key)`tls.client_certs[].key` A plain text certificate key to use. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ### [](#tls-client_certs-key_file)`tls.client_certs[].key_file` The path of a certificate key to use. **Type**: `string` **Default**: `""` ### [](#tls-client_certs-password)`tls.client_certs[].password` A plain text password for when the private key is password encrypted in PKCS#1 or PKCS#8 format. The obsolete `pbeWithMD5AndDES-CBC` algorithm is not supported for the PKCS#8 format. Because the obsolete pbeWithMD5AndDES-CBC algorithm does not authenticate the ciphertext, it is vulnerable to padding oracle attacks that can let an attacker recover the plaintext. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: password: foo # --- password: ${KEY_PASSWORD} ``` ### [](#tls-enable_renegotiation)`tls.enable_renegotiation` Whether to allow the remote server to request renegotiation. Enable this option if you’re seeing the error message `local error: tls: no renegotiation`. Requires version 3.45.0 or later. **Type**: `bool` **Default**: `false` ### [](#tls-enabled)`tls.enabled` Whether to use TLS for the connection to the Redpanda cluster. **Type**: `bool` **Default**: `false` ### [](#tls-root_cas)`tls.root_cas` Specify a root certificate authority to use (optional). This is a string that represents a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for inline certificate data or `root_cas_file` for file-based certificate loading. > ⚠️ **CAUTION** > > This field contains sensitive information that usually shouldn’t be added to a configuration directly. For more information, see [Secrets](../../../configuration/secrets/). **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas: |- -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- ``` ### [](#tls-root_cas_file)`tls.root_cas_file` Specify the path to a root certificate authority file (optional). This is a file, often with a `.pem` extension, which contains a certificate chain from the parent-trusted root certificate, through possible intermediate signing certificates, to the host certificate. Use either this field for file-based certificate loading or `root_cas` for inline certificate data. **Type**: `string` **Default**: `""` ```yaml # Examples: root_cas_file: ./root_cas.pem ``` ### [](#tls-skip_cert_verify)`tls.skip_cert_verify` Whether to skip server-side certificate verification. Set to `true` only for testing environments as this reduces security by disabling certificate validation. When using self-signed certificates or in development, this may be necessary, but should never be used in production. Consider using `root_cas` or `root_cas_file` to specify trusted certificates instead of disabling verification entirely. **Type**: `bool` **Default**: `false` ### [](#topic)`topic` The Redpanda topic where trace data is written. This topic should be dedicated to traces and configured with appropriate retention policies. Default: `otel-traces` **Type**: `string` **Default**: `otel-traces` --- # Page 328: Configuration **URL**: https://docs.redpanda.com/redpanda-connect/configuration/about.md --- # Configuration --- title: Configuration latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: about page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/about.adoc description: Learn about different options for configuring Redpanda Connect. page-git-created-date: "2024-05-24" page-git-modified-date: "2025-10-21" --- Redpanda Connect pipelines are configured in a YAML file that consists of a number of root sections, arranged like so: #### Common ```yaml input: kafka: addresses: [ TODO ] topics: [ foo, bar ] consumer_group: foogroup pipeline: processors: - mapping: | root.message = this root.meta.link_count = this.links.length() output: aws_s3: bucket: TODO path: '${! meta("kafka_topic") }/${! json("message.id") }.json' ``` #### Full ```yaml http: address: 0.0.0.0:4195 debug_endpoints: false input: kafka: addresses: [ TODO ] topics: [ foo, bar ] consumer_group: foogroup buffer: none: {} pipeline: processors: - mapping: | root.message = this root.meta.link_count = this.links.length() output: aws_s3: bucket: TODO path: '${! meta("kafka_topic") }/${! json("message.id") }.json' input_resources: [] cache_resources: [] processor_resources: [] rate_limit_resources: [] output_resources: [] logger: level: INFO static_fields: '@service': benthos metrics: prometheus: {} tracer: none: {} shutdown_timeout: 20s shutdown_delay: "" ``` Most sections represent a component type, which you can read about in more detail in [this document](../../components/about/). These types are hierarchical. For example, an `input` can have a list of child `processor` types attached to it, which in turn can have their own `processor` children. This is powerful but can potentially lead to large and cumbersome configuration files. This document outlines tooling provided by Redpanda Connect to help with writing and managing these more complex configuration files. ## [](#testing)Testing For guidance on how to write and run unit tests for your configuration files read [this guide](../unit_testing/). ## [](#customizing-your-configuration)Customizing your configuration Sometimes it’s useful to write a configuration where certain fields can be defined during deployment. For this purpose Redpanda Connect supports [environment variable interpolation](../interpolation/), allowing you to set fields in your config with environment variables like so: ```yaml input: kafka: addresses: - ${KAFKA_BROKER:localhost:9092} topics: - ${KAFKA_TOPIC:default-topic} ``` This is very useful for sharing configuration files across different deployment environments. ## [](#labels)Labels Labels are unique, user-defined identifiers used throughout Redpanda Connect configurations. They serve two purposes: - **Reference:** Allow different parts of your pipeline to refer to specific components or resources. - **Readability:** Make your configuration more understandable for humans, especially in complex deployments. You can assign labels to most pipeline components, including resources, inputs, outputs, processors, and entire pipelines. Using clear, descriptive labels improves both maintainability and clarity. Labels are commonly applied to the following components: ### [](#resources)Resources Labels identify [reusable resources](#reuse) such as processors, caches, and rate limiters, making them easy to reference elsewhere in your pipeline. ```yaml processor_resources: - label: my-transformer # Processor resource label mapping: 'root = content().uppercase()' cache_resources: - label: user-cache # Cache resource label memory: default_ttl: 300s rate_limit_resources: - label: api-limiter # Rate limiter resource label local: count: 100 interval: 1m ``` ### [](#pipelines-streams-mode)Pipelines (streams mode) When running in [streams mode](../../guides/streams_mode/about/) with the Streams API, labels serve as pipeline names. This enables you to create and reference pipelines by their label through the API. ```bash # Create a pipeline labeled "data-processor" curl -X POST http://localhost:4195/streams/data-processor \ -H "Content-Type: application/yaml" \ -d @pipeline-config.yaml # Reference a pipeline by its label curl http://localhost:4195/streams/data-processor ``` ### [](#component-labeling-for-clarity)Component labeling for clarity You can also use labels on inputs, outputs, processors, and other components to improve the human-readability of your configuration and make troubleshooting easier. For example: ```yaml input: label: ingest_api http_server: {} pipeline: label: user_data_ingest processors: - label: sanitize_fields mapping: 'root = this.trim()' - resource: my-transformer ``` ## [](#label-naming-requirements)Label naming requirements Labels must meet the following criteria: - **Length**: 3-128 characters - **Allowed characters**: Alphanumeric, hyphens, and underscores (`A-Za-z0-9-_`) - **Case sensitivity**: Labels are case-sensitive Example valid labels my-processor data\_transformer\_01 UserAnalytics-v2 Example invalid labels ab // Too short (less than 3 characters) my.processor // Invalid character: period my processor // Invalid character: space ## [](#reuse)Reusing configuration snippets Sometimes it’s necessary to use a rather large component multiple times. Instead of copy/pasting the configuration or using YAML anchors you can define your component [as a resource](../resources/). In the following example we want to make an HTTP request with our payloads. Occasionally the payload might get rejected due to garbage within its contents, and so we catch these rejected requests, attempt to "cleanse" the contents and try to make the same HTTP request again. Since the HTTP request component is quite large (and likely to change over time) we make sure to avoid duplicating it by defining it as a resource `get_foo`: ```yaml pipeline: processors: - resource: get_foo - catch: - mapping: | root = this root.content = this.content.strip_html() - resource: get_foo processor_resources: - label: get_foo http: url: http://example.com/foo verb: POST headers: SomeThing: "set-to-this" SomeThingElse: "set-to-something-else" ``` ### [](#feature-toggles)Feature toggles Resources can be imported separately to your config file with the cli flag `-r` or `-resources`, which is a useful way to switch out resources with common names based on your chosen environment. For example, with a main configuration file `config.yaml`: ```yaml pipeline: processors: - resource: get_foo ``` And then two resource files, one stored at the path `./staging/request.yaml`: ```yaml processor_resources: - label: get_foo http: url: http://example.com/foo verb: POST headers: SomeThing: "set-to-this" SomeThingElse: "set-to-something-else" ``` And another stored at the path `./production/request.yaml`: ```yaml processor_resources: - label: get_foo http: url: http://example.com/bar verb: PUT headers: Desires: "are-empty" ``` We can select our chosen resource by changing which file we import, either running: ```bash rpk connect run -r ./staging/request.yaml ./config.yaml ``` Or: ```bash rpk connect run -r ./production/request.yaml ./config.yaml ``` These flags also support wildcards, which allows you to import an entire directory of resource files like `rpk connect run -r "./staging/*.yaml" ./config.yaml`. You can find out more about configuration resources in the [resources document](../resources/). ### [](#templating)Templating Resources can only be instantiated with a single configuration, which means they aren’t suitable for cases where the configuration is required in multiple places but with slightly different parameters. Redpanda Connect has a (currently experimental) alternative feature called templates, with which it’s possible to define a custom configuration schema and a template for building a configuration from that schema. You can read more about templates [in this guide](../templating/). ## [](#reloading)Reloading It’s possible to have a running instance of Redpanda Connect reload configurations, including resource files imported with `-r`/`--resources`, automatically when the files are updated without needing to manually restart the service. This is done by specifying the `-w`/`--watcher` flag when running Redpanda Connect in normal mode or in streams mode: ```bash # Normal mode rpk connect run -w -r ./production/request.yaml ./config.yaml ``` ```bash # Streams mode rpk connect streams -w -r ./production/request.yaml ./stream_configs/*.yaml ``` If a file update results in configuration parsing or linting errors then the change is ignored (with logs informing you of the problem) and the previous configuration will continue to be run (until the issues are fixed). ## [](#enabling-discovery)Enabling discovery The discoverability of configuration fields is a common headache with any configuration driven application. The classic solution is to provide curated documentation that is often hosted on a dedicated site. However, a user often only needs to get their hands on a short, runnable example config file for their use case. They just need to see the format and field names as the fields themselves are usually self explanatory. Forcing such a user to navigate a website, scrolling through paragraphs of text, seems inefficient when all they actually needed to see was something like: ```yaml input: amqp_0_9: urls: [ amqp://guest:guest@localhost:5672/ ] consumer_tag: benthos-consumer queue: benthos-queue prefetch_count: 10 prefetch_size: 0 output: stdout: {} ``` In order to make this process easier Redpanda Connect is able to generate usable configuration examples for any types, and you can do this from the binary using the `create` subcommand. If, for example, we wanted to generate a config with a websocket input, a Kafka output and a [`mapping` processor](../../components/processors/mapping/) in the middle, we could do it with the following command: ```bash rpk connect create websocket/mapping/kafka ``` > 💡 **TIP** > > To see which components Redpanda Connect offers, use `rpk connect list`. All of these generated configuration examples also include other useful config sections such as `metrics`, `logging`, etc with sensible defaults. For more information read the output from `rpk connect create --help`. ## [](#help-with-debugging)Help with debugging Once you have a config written you now move onto the next headache of proving that it works, and understanding why it doesn’t. Redpanda Connect, like most good config driven services, performs validation on configs and tries to provide sensible error messages. However, with validation it can be hard to capture all problems, and the user usually understands their intentions better than the service. In order to help expose and diagnose config errors Redpanda Connect provides two mechanisms, linting and echoing. ### [](#linting)Linting If you attempt to run a config that has linting errors Redpanda Connect will print the errors and halt execution. If, however, you want to test your configs before deployment you can do so with the `lint` subcommand: For example, imagine we have a config `foo.yaml`, where we intend to read from AMQP, but there is a typo in our config struct: ```text input: amqp_0_9: yourl: amqp://guest:guest@rabbitmqserver:5672/ ``` We can catch this error before attempting to run the config: ```bash rpk connect lint ./foo.yaml ./foo.yaml: line 3: field yourl not recognized ``` For more information read the output from `rpk connect lint --help`. ### [](#echoing)Echoing Echoing is where Redpanda Connect can print back your configuration _after_ it has been parsed. It is done with the `echo` subcommand, which is able to show you a normalized version of your config, allowing you to see how it was interpreted: ```bash rpk connect echo ./your-config.yaml ``` You can check the output of the above command to see if certain sections are missing or fields are incorrect, which allows you to pinpoint typos in the config. ## [](#shutting-down)Shutting down Under normal operating conditions, the Redpanda Connect process will shut down when there are no more messages produced by inputs and the final message has been processed. The shutdown procedure can also be initiated by sending the process a interrupt (`SIGINT`) or termination (`SIGTERM`) signal. There are two top-level configuration options that control the shutdown behavior: `shutdown_timeout` and `shutdown_delay`. ### [](#shutdown-delay)Shutdown delay The `shutdown_delay` option can be used to delay the start of the shutdown procedure. This is useful for pipelines that need a short grace period to have their metrics and traces scraped. While the shutdown delay is in effect, the HTTP metrics endpoint continues to be available for scraping and any active tracers are free to flush remaining traces. The shutdown delay can be interrupted by sending the Redpanda Connect process a second OS interrupt or termination signal. ### [](#shutdown-timeout)Shutdown timeout The `shutdown_timeout` option sets a hard deadline for Redpanda Connect process to gracefully terminate. If this duration is exceeded then the process is forcefully terminated and any messages that were in-flight will be dropped. This option takes effect after the `shutdown_delay` duration has passed if that is enabled. --- # Page 329: Configure an Allow or Deny List **URL**: https://docs.redpanda.com/redpanda-connect/configuration/allow_and_deny_lists.md --- # Configure an Allow or Deny List --- title: Configure an Allow or Deny List latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: allow_and_deny_lists page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: allow_and_deny_lists.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/allow_and_deny_lists.adoc description: Learn how to configure an allow or deny list for a Redpanda Connect instance. page-git-created-date: "2024-11-19" page-git-modified-date: "2025-08-08" --- > 📝 **NOTE** > > This feature requires an [enterprise license](../../get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that’s valid for 30 days. You can configure an allow or deny list to limit the Redpanda Connect components that users can run within data pipelines on a Redpanda Connect instance. Introduced in version 4.39.0. ## [](#limitations)Limitations - You can configure only one allow list, or one deny list per Redpanda Connect instance. - When you add the name of a component to your allow or deny list, you grant or block access to all related components. For example, if you include `aws_s3` in your list, you grant or block access to the entire `aws_s3` family of components, including the cache, input, and output. ## [](#configure-an-allow-list)Configure an allow list To configure an allow list: 1. Create a `connector_list.yaml` file in the following location: `/etc/redpanda/connector_list.yaml`. 2. In the [Connector Catalog](../../components/about/), find the exact name of the component family you want to grant access to. For example: `aws_s3` and `azure_cosmosdb`. 3. Add the names of the component families to the `connector_list.yaml` file, using the following format: ```yaml allow: - aws_s3 - azure_cosmosdb ``` 4. Save your configuration. ## [](#configure-a-deny-list)Configure a deny list To configure a deny list: 1. Create a `connector_list.yaml` file in the following location: `/etc/redpanda/connector_list.yaml`. 2. In the [Connector Catalog](../../components/about/), find the exact name of the component family you want to block access to. For example: `aws_s3` and `azure_cosmosdb`. 3. Add the names of the component families to the `connector_list.yaml` file, using the following format: ```yaml deny: - aws_s3 - azure_cosmosdb ``` 4. Save your configuration. --- # Page 330: Message Batching **URL**: https://docs.redpanda.com/redpanda-connect/configuration/batching.md --- # Message Batching --- title: Message Batching latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: batching page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: batching.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/batching.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- Redpanda Connect is able to join sources and sinks with sometimes conflicting batching behaviors without sacrificing its strong delivery guarantees. It’s also able to perform powerful [processing functions](../windowed_processing/) across batches of messages such as grouping, archiving and reduction. Therefore, batching within Redpanda Connect is a mechanism that serves multiple purposes: 1. [Performance (throughput)](#performance) 2. [Grouped message processing](#grouped-message-processing) 3. [Compatibility (mixing multi and single part message protocols)](#compatibility) ## [](#performance)Performance For most users the only benefit of batching messages is improving throughput over your output protocol. For some protocols this can happen in the background and requires no configuration from you. However, if an output has a `batching` configuration block this means it benefits from batching and requires you to specify how you’d like your batches to be formed by configuring a [batching policy](#batch-policy): ```yaml output: kafka: addresses: [ todo:9092 ] topic: benthos_stream # Either send batches when they reach 10 messages or when 100ms has passed # since the last batch. batching: count: 10 period: 100ms ``` However, a small number of inputs such as [`kafka`](../../components/inputs/kafka/) must be consumed sequentially (in this case by partition) and therefore benefit from specifying your batch policy at the input level instead: ```yaml input: kafka: addresses: [ todo:9092 ] topics: [ benthos_input_stream ] batching: count: 10 period: 100ms output: kafka: addresses: [ todo:9092 ] topic: benthos_stream ``` Inputs that behave this way are documented as such and have a `batching` configuration block. Sometimes you may prefer to create your batches before processing in order to benefit from [batch wide processing](#grouped-message-processing), in which case if your input doesn’t already support [a batch policy](#batch-policy) you can instead use a [`broker`](../../components/inputs/broker/), which also allows you to combine inputs with a single batch policy: ```yaml input: broker: inputs: - resource: foo - resource: bar batching: count: 50 period: 500ms ``` This also works the same with [output brokers](../../components/outputs/broker/). ## [](#grouped-message-processing)Grouped message processing And some processors such as [`while`](../../components/processors/while/) are executed once across a whole batch, you can avoid this behavior with the [`for_each` processor](../../components/processors/for_each/): ```yaml pipeline: processors: - for_each: - while: at_least_once: true max_loops: 0 check: errored() processors: - catch: [] # Wipe any previous error - resource: foo # Attempt this processor until success ``` There’s a vast number of processors that specialise in operations across batches such as [grouping](../../components/processors/group_by/) and [archiving](../../components/processors/archive/). For example, the following processors group a batch of messages according to a metadata field and compresses them into separate `.tar.gz` archives: ```yaml pipeline: processors: - group_by_value: value: ${! meta("kafka_partition") } - archive: format: tar - compress: algorithm: gzip output: aws_s3: bucket: TODO path: docs/${! meta("kafka_partition") }/${! count("files") }-${! timestamp_unix_nano() }.tar.gz ``` For more examples of batched (or windowed) processing check out [this document](../windowed_processing/). ## [](#compatibility)Compatibility Redpanda Connect is able to read and write over protocols that support multiple part messages, and all payloads travelling through Redpanda Connect are represented as a multiple part message. Therefore, all components within Redpanda Connect are able to work with multiple parts in a message as standard. When messages reach an output that _doesn’t_ support multiple parts the message is broken down into an individual message per part, and then one of two behaviors happen depending on the output. If the output supports batch sending messages then the collection of messages are sent as a single batch. Otherwise, Redpanda Connect falls back to sending the messages sequentially in multiple, individual requests. This behavior means that not only can multiple part message protocols be easily matched with single part protocols, but also the concept of multiple part messages and message batches are interchangeable within Redpanda Connect. ### [](#shrinking-batches)Shrinking batches A message batch (or multiple part message) can be broken down into smaller batches using the [`split`](../../components/processors/split/) processor: ```yaml input: # Consume messages that arrive in three parts. resource: foo processors: # Drop the third part - select_parts: parts: [ 0, 1 ] # Then break our message parts into individual messages - split: size: 1 ``` This is also useful when your input source creates batches that are too large for your output protocol: ```yaml input: aws_s3: bucket: todo pipeline: processors: - decompress: algorithm: gzip - unarchive: format: tar # Limit batch sizes to 5MB - split: byte_size: 5_000_000 ``` ## [](#batch-policy)Batch policy When an input or output component has a config field `batching` that means it supports a batch policy. This is a mechanism that allows you to configure exactly how your batching should work on messages before they are routed to the input or output it’s associated with. Batches are considered complete and will be flushed downstream when either of the following conditions are met: - The `byte_size` field is non-zero and the total size of the batch in bytes matches or exceeds it (disregarding metadata.) - The `count` field is non-zero and the total number of messages in the batch matches or exceeds it. - A message added to the batch causes the [`check`](../../guides/bloblang/about/) to return to `true`. - The `period` field is non-empty and the time since the last batch exceeds its value. This allows you to combine conditions: ```yaml output: kafka: addresses: [ todo:9092 ] topic: benthos_stream # Either send batches when they reach 10 messages or when 100ms has passed # since the last batch. batching: count: 10 period: 100ms ``` > ⚠️ **CAUTION** > > A batch policy has the capability to _create_ batches, but not to break them down. If your configured pipeline is processing messages that are batched _before_ they reach the batch policy then they may circumvent the conditions you’ve specified here, resulting in sizes you aren’t expecting. If you are affected by this limitation then consider breaking the batches down with a [`split` processor](../../components/processors/split/) before they reach the batch policy. ### [](#post-batch-processing)Post-batch processing A batch policy also has a field `processors` which allows you to define an optional list of [processors](../../components/processors/about/) to apply to each batch before it is flushed. This is a good place to aggregate or archive the batch into a compatible format for an output: ```yaml output: http_client: url: http://localhost:4195/post batching: count: 10 processors: - archive: format: lines ``` The above config will batch up messages and then merge them into a line delimited format before sending it over HTTP. This is an easier format to parse than the default which would have been [rfc1342](https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html). During shutdown any remaining messages waiting for a batch to complete will be flushed down the pipeline. --- # Page 331: Dynamic Inputs and Outputs **URL**: https://docs.redpanda.com/redpanda-connect/configuration/dynamic_inputs_and_outputs.md --- # Dynamic Inputs and Outputs --- title: Dynamic Inputs and Outputs latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: dynamic_inputs_and_outputs page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: dynamic_inputs_and_outputs.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/dynamic_inputs_and_outputs.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- It is possible to have sets of inputs and outputs in Redpanda Connect that can be added, updated and removed during runtime with the [dynamic fan in](../../components/inputs/dynamic/) and [dynamic fan out](../../components/outputs/dynamic/) types. Dynamic inputs and outputs are each identified by unique string labels, which are specified when adding them either in configuration or via the HTTP API. The labels are useful when querying which types are active. ## [](#api)API The API for dynamic types (both inputs and outputs) is a collection of HTTP REST endpoints: ### [](#inputs)`/inputs` Returns a JSON object that maps input labels to an object containing details about the input, including uptime and configuration. If the input has terminated naturally the uptime will be set to `stopped`. ```json { "": { "uptime": "", "config": }, ... } ``` ### [](#inputsinput_label)`/inputs/{input_label}` GET returns the configuration of the input idenfified by `input_label`. POST sets the input `input_label` to the body of the request parsed as a JSON configuration. If the input label already exists the previous input is first stopped and removed. DELETE stops and removes the input identified by `input_label`. ### [](#outputs)`/outputs` Returns a JSON object that maps output labels to an object containing details about the output, including uptime and configuration. If the output has terminated naturally the uptime will be set to `stopped`. ```json { "": { "uptime": "", "config": }, ... } ``` ### [](#outputsoutput_label)`/outputs/{output_label}` GET returns the configuration of the output idenfified by `output_label`. POST sets the output `output_label` to the body of the request parsed as a JSON configuration. If the output label already exists the previous output is first stopped and removed. DELETE stops and removes the output identified by `output_label`. A custom prefix can be set for these endpoints in configuration. ## [](#applications)Applications Dynamic types are useful when a platforms data streams might need to change regularly and automatically. It is also useful for triggering batches of platform data, e.g. a cron job can be created to send hourly curl requests that adds a dynamic input to read a file of sample data: ```sh curl http://localhost:4195/inputs/read_sample -d @- << EOF { "file": { "path": "/tmp/line_delim_sample_data.txt" } } EOF ``` Some inputs have a finite lifetime, e.g. `s3` without an SQS queue configured will close once the whole bucket has been read. When a dynamic types lifetime ends the `uptime` field of an input listing will be set to `stopped`. You can use this to write tools that trigger new inputs (to move onto the next bucket, for example). --- # Page 332: Error Handling **URL**: https://docs.redpanda.com/redpanda-connect/configuration/error_handling.md --- # Error Handling --- title: Error Handling latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: error_handling page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: error_handling.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/error_handling.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2025-03-27" --- Redpanda Connect supports a range of [processors](../../components/processors/about/), such as `http` and `aws_lambda`, that may fail when retry attempts are exhausted. When a processor fails, the message data continues through the pipeline mostly unchanged, except for the addition of a metadata flag, which you can use for handling errors. This topic explains some common error-handling patterns, including dropping messages, recovering them with more processing, and routing them to a dead-letter queue. It also shows how to combine these approaches, where appropriate, and provides a method for [trapping unhandled errors](#logging-fatal-errors). ## [](#abandon-on-failure)Abandon on failure You can use the [`try` processor](../../components/processors/try/) to define a list of processors that are executed in sequence. If a processor fails for a particular message, that message skips the remaining processors. For example: - If `processor_1` fails to process a message, that message skips `processor_2` and `processor_3`. - If a message is processed by `processor_1`, but `processor_2` fails, that message skips `processor_3`, and so on. ```yaml pipeline: processors: - try: - resource: processor_1 - resource: processor_2 # Skip if processor_1 fails - resource: processor_3 # Skip if processor_1 or processor_2 fails ``` ## [](#recover-failed-messages)Recover failed messages You can also route failed messages through defined processing steps using a [`catch` processor](../../components/processors/catch/). For example, if `processor_1` fails to process a message, it is rerouted to `processor_2`. ```yaml pipeline: processors: - resource: processor_1 # Processor that might fail - catch: - resource: processor_2 # Processes rerouted messages ``` After messages complete all processing steps defined in the `catch` block, failure flags are removed and they are treated like regular messages. To keep failure flags in messages, you can simulate a `catch` block using a [`switch` processor](../../components/processors/switch/): ```yaml pipeline: processors: - resource: processor_1 # Processor that might fail - switch: - check: errored() processors: - resource: processor_2 # Processes rerouted messages ``` ## [](#logging-errors)Logging errors When an error occurs, there may be useful information stored in the error flag. You can use [`error`](../../guides/bloblang/functions/#error) Bloblang function interpolations to write this information to logs. You can also add the following Bloblang functions to expose additional details about the processor that triggered the error. - [`error_source_label`](../../guides/bloblang/functions/#error_source_label) - [`error_source_name`](../../guides/bloblang/functions/#error_source_name) - [`error_source_path`](../../guides/bloblang/functions/#error_source_path) For example, this configuration catches processor failures and writes the following information to logs: - The label of the processor (`${!error_source_label()}`) that failed - The cause of the failure (`${!error()}`) ```yaml pipeline: processors: - try: - resource: processor_1 # Processor that might fail - resource: processor_2 # Processor that might fail - resource: processor_3 # Processor that might fail - catch: - log: message: "Processor ${!error_source_label()} failed due to: ${!error()}" ``` You could also add an error message to the message payload: ```yaml pipeline: processors: - resource: processor_1 # Processor that might fail - resource: processor_2 # Processor that might fail - resource: processor_3 # Processor that might fail - catch: - mapping: | root = this root.meta.error = error() ``` ### [](#logging-fatal-errors)Logging fatal errors During development and testing, you can use the [`crash` processor](../../components/processors/crash/) to halt pipeline execution and log a fatal message when an unhandled error occurs. Use Bloblang functions to customize the message. For example: ```yaml pipeline: processors: - resource: processor_1 # Processor that might fail - catch: # Writes a custom fatal log message - crash: "Processing failed at ${!error_source_label()} due to: ${!error()}" ``` ## [](#attempt-until-success)Attempt until success To process a particular message until it is successful, try using a [`retry`](../../components/processors/retry/) processor: ```yaml pipeline: processors: - retry: backoff: initial_interval: 1s max_interval: 5s max_elapsed_time: 30s processors: # Retries this processor until the message is processed, or the maximum elapsed time is reached. - resource: processor_1 ``` ## [](#drop-failed-messages)Drop failed messages To filter out any failed messages from your pipeline, you can use a [`mapping` processor](../../components/processors/mapping/): ```yaml pipeline: processors: - mapping: root = if errored() { deleted() } ``` The mapping uses the error flag to identify any failed messages in a batch and drops the messages, which propagates acknowledgements (also known as "acks") upstream to the pipeline’s input. ## [](#reject-messages)Reject messages Some inputs, such as `nats`, `gcp_pubsub`, and `amqp_1`, support nacking (rejecting) messages. Rather than delivering unprocessed messages to your output, you can use the [`reject_errored` output](../../components/outputs/reject_errored/) to perform a nack (or rejection) on them: ```yaml output: reject_errored: resource: processor_1 # Only non-errored messages go here ``` ## [](#route-to-a-dead-letter-queue)Route to a dead-letter queue You can also route failed messages to a different output by nesting the [`reject_errored` output](../../components/outputs/reject_errored/) within a [`fallback` output](../../components/outputs/fallback/) ```yaml output: fallback: - reject_errored: resource: processor_1 # Only non-errored messages go here - resource: processor_2 # Only errored messages, or delivery failures to processor_1, go here ``` If you want to route data differently based on the type of error message, you can use a [`switch` output](../../components/outputs/switch/): ```yaml output: switch: cases: # Capture specifically cat-related errors - check: errored() && error().contains("meow") output: resource: processor_1 # Capture all other errors - check: errored() output: resource: processor_2 # Finally, route all successfully processed messages here - output: resource: processor_3 ``` Finally, you can attach additional metadata when routing messages to the dead-letter queue, such as the error message. This can be done by running a series of [processors](../../components/processors/about/) before sending the data to the final [output](../../components/outputs/about/). ```yaml output: fallback: - reject_errored: resource: processor_1 # Only non-errored messages go here - processors: - mutation: | root.error = @fallback_error # Adds the error message before sending the message to the dead-letter queue output resource: processor_2 # Only errored messages, or delivery failures to processor_1, go here ``` --- # Page 333: Field Paths **URL**: https://docs.redpanda.com/redpanda-connect/configuration/field_paths.md --- # Field Paths --- title: Field Paths latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: field_paths page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: field_paths.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/field_paths.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- Many components within Redpanda Connect allow you to target certain fields using a JSON dot path. The syntax of a path within Redpanda Connect is similar to [JSON Pointers](https://tools.ietf.org/html/rfc6901), except with dot separators instead of slashes (and no leading dot.) When a path is used to set a value any path segment that does not yet exist in the structure is created as an object. For example, if we had the following JSON structure: ```json { "foo": { "bar": 21 } } ``` The query path `foo.bar` would return `21`. The characters `~` (%x7E) and `.` (%x2E) have special meaning in Redpanda Connect paths. Therefore `~` needs to be encoded as `~0` and `.` needs to be encoded as `~1` when these characters appear within a key. For example, if we had the following JSON structure: ```json { "foo.foo": { "bar~bo": { "": { "baz": 22 } } } } ``` The query path `foo~1foo.bar~0bo..baz` would return `22`. ## [](#arrays)Arrays When Redpanda Connect encounters an array while traversing a JSON structure it requires the next path segment to be either an integer of an existing index or, depending on whether the path is used to query or set the target value, the character `*` or `-` respectively. For example, if we had the following JSON structure: ```json { "foo": [ 0, 1, { "bar": 23 } ] } ``` The query path `foo.2.bar` would return `23`. ### [](#querying)Querying When a query reaches an array the character `*` indicates that the query should return the value of the remaining path from each array element (within an array.) ### [](#setting)Setting When an array is reached the character `-` indicates that a new element should be appended to the end of the existing elements, if this character is not the final segment of the path then an object is created. --- # Page 334: Interpolation **URL**: https://docs.redpanda.com/redpanda-connect/configuration/interpolation.md --- # Interpolation --- title: Interpolation latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: interpolation page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: interpolation.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/interpolation.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-02-06" --- Redpanda Connect allows you to dynamically set config fields with environment variables anywhere within a config file using the syntax `${}` (or `${:}` in order to specify a default value). This is useful for setting environment specific fields such as addresses: ```yaml input: kafka: addresses: [ "${BROKERS}" ] consumer_group: redpanda_connect_consumer topics: [ "haha_business" ] ``` ```sh BROKERS="foo:9092,bar:9092" rpk connect run ./config.yaml ``` If a literal string is required that matches this pattern (`${foo}`) you can escape it with double brackets. For example, the string `${{foo}}` is read as the literal `${foo}`. ## [](#undefined-variables)Undefined variables When an environment variable interpolation is found within a config, does not have a default value specified, and the environment variable is not defined a linting error will be reported. In order to avoid this it is possible to specify environment variable interpolations with an explicit empty default value by adding the colon without a following value, i.e. `${FOO:}` would be equivalent to `${FOO}` and would not trigger a linting error should `FOO` not be defined. ## [](#yaml-tags)YAML tags By default, Redpanda Connect interpolates environment variables as strings. You can use [YAML tags](https://yaml.org/spec/1.2.2/#24-tags) to interpret values as another scalar type, such as integers. ```yaml output: redpanda: # ... batching: count: !!int ${BATCHING_COUNT:500} period: "${BATCHING_PERIOD:1s}" ``` Redpanda Connect supports the [core schema tags](https://yaml.org/spec/1.2.2/#103-core-schema) for scalar types: - `null` - `bool` - `int` - `float` - `str` (default) ## [](#bloblang-queries)Bloblang queries Some Redpanda Connect fields also support [Bloblang](../../guides/bloblang/about/) function interpolations, which are much more powerful expressions that allow you to query the contents of messages and perform arithmetic. The syntax of a function interpolation is `${!}`, where the contents are a bloblang query (the right-hand-side of a bloblang map) including a range of [functions](../../guides/bloblang/about/#functions). For example, with the following config: ```yaml output: kafka: addresses: [ "TODO:6379" ] topic: 'dope-${! json("topic") }' ``` A message with the contents `{"topic":"foo","message":"hello world"}` would be routed to the Kafka topic `dope-foo`. If a literal string is required that matches this pattern (`${!foo}`) then, similar to environment variables, you can escape it with double brackets. For example, the string `${{!foo}}` would be read as the literal `${!foo}`. Bloblang supports arithmetic, boolean operators, coalesce and mapping expressions. For more in-depth details about the language [check out the docs](../../guides/bloblang/about/). ## [](#examples)Examples ### [](#reference-metadata)Reference metadata A common usecase for interpolated functions is dynamic routing at the output level using metadata: ```yaml output: kafka: addresses: [ TODO ] topic: ${! meta("output_topic") } key: ${! meta("key") } ``` ### [](#coalesce-and-mapping)Coalesce and mapping Bloblang supports coalesce and mapping, which makes it easy to extract values from slightly varying data structures: ```yaml pipeline: processors: - cache: resource: foocache operator: set key: '${! json().message.(foo | bar).id }' value: '${! content() }' ``` Here’s a map of inputs to resulting values: {"foo":{"a":{"baz":"from\_a"},"c":{"baz":"from\_c"}}} -> from\_a {"foo":{"b":{"baz":"from\_b"},"c":{"baz":"from\_c"}}} -> from\_b {"foo":{"b":null,"c":{"baz":"from\_c"}}} -> from\_c ### [](#delayed-processing)Delayed processing We have a stream of JSON documents each with a unix timestamp field `doc.received_at` which is set when our platform receives it. We wish to only process messages an hour _after_ they were received. We can achieve this by running the `sleep` processor using an interpolation function to calculate the seconds needed to wait for: ```yaml pipeline: processors: - sleep: duration: '${! 3600 - ( timestamp_unix() - json("doc.created_at").number() ) }s' ``` If the calculated result is less than or equal to zero the processor does not sleep at all. If the value of `doc.created_at` is a string then our method `.number()` will attempt to parse it into a number. --- # Page 335: Metadata **URL**: https://docs.redpanda.com/redpanda-connect/configuration/metadata.md --- # Metadata --- title: Metadata latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: metadata page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: metadata.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/metadata.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-02-04" --- In Redpanda Connect each message has raw contents and metadata, which is a map of key/value pairs representing an arbitrary amount of complementary data. When an input protocol supports attributes or metadata they will automatically be added to your messages, refer to the respective input documentation for a list of metadata keys. When an output supports attributes or metadata any metadata key/value pairs in a message will be sent (subject to service limits). ## [](#editing-metadata)Editing metadata Redpanda Connect allows you to add and remove metadata using the [`mapping` processor](../../components/processors/mapping/). For example, you can do something like this in your pipeline: ```yaml pipeline: processors: - mapping: | # Remove all existing metadata from messages meta = deleted() # Add a new metadata field `time` from the contents of a JSON # field `event.timestamp` meta time = event.timestamp ``` You can also use [Bloblang](../../guides/bloblang/about/) to delete individual metadata keys with: ```bloblang meta foo = deleted() ``` Or do more interesting things like remove all metadata keys with a certain prefix: ```bloblang meta = @.filter(kv -> !kv.key.has_prefix("kafka_")) ``` ## [](#using-metadata)Using metadata Metadata values can be referenced in any field that supports [interpolation functions](../interpolation/). For example, you can route messages to Kafka topics using interpolation of metadata keys: ```yaml output: kafka: addresses: [ TODO ] topic: ${! meta("target_topic") } ``` Redpanda Connect also allows you to conditionally process messages based on their metadata with the [`switch` processor](../../components/processors/switch/): ```yaml pipeline: processors: - switch: - check: '@doc_type == "nested"' processors: - sql_insert: driver: mysql dsn: foouser:foopassword@tcp(localhost:3306)/foodb table: footable columns: [ foo, bar, baz ] args_mapping: | root = [ this.document.foo, this.document.bar, @kafka_topic, ] # In: {"document":{"foo":"value1","bar":"value2"}} ``` ## [](#restricting-metadata)Restricting metadata Outputs that support metadata, headers or some other variant of enriched fields on messages will attempt to send all metadata key/value pairs by default. However, sometimes it’s useful to refer to metadata fields at the output level even though we do not wish to send them with our data. In this case it’s possible to restrict the metadata keys that are sent with the field `metadata.exclude_prefixes` within the respective output config. For example, if we were sending messages to kafka using a metadata key `target_topic` to determine the topic but we wished to prevent that metadata key from being sent as a header we could use the following configuration: ```yaml output: kafka: addresses: [ TODO ] topic: ${! meta("target_topic") } metadata: exclude_prefixes: - target_topic ``` And when the list of metadata keys that we do _not_ want to send is large it can be helpful to use a [Bloblang mapping](../../guides/bloblang/about/) in order to give all of these "private" keys a common prefix: ```yaml pipeline: processors: # Has an explicit list of public metadata keys, and everything else is given # an underscore prefix. - mapping: | let allowed_meta = [ "foo", "bar", "baz", ] meta = @.map_each_key(key -> if !$allowed_meta.contains(key) { "_" + key }) output: kafka: addresses: [ TODO ] topic: ${! meta("_target_topic") } metadata: exclude_prefixes: [ "_" ] ``` --- # Page 336: Processing Pipelines **URL**: https://docs.redpanda.com/redpanda-connect/configuration/processing_pipelines.md --- # Processing Pipelines --- title: Processing Pipelines latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: processing_pipelines page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: processing_pipelines.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/processing_pipelines.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- Within a Redpanda Connect configuration, in between `input` and `output`, is a `pipeline` section. This section describes an array of [processors](../../components/processors/about/) that are to be applied to _all_ messages, and are not bound to any particular input or output. If you have processors that are heavy on CPU and aren’t specific to a certain input or output they are best suited for the pipeline section. It is advantageous to use the pipeline section as it allows you to set an explicit number of parallel threads of execution: ```yaml input: resource: foo pipeline: threads: 4 processors: - mapping: | root = this fans = fans.map_each(match { this.obsession > 0.5 => this _ => deleted() }) output: resource: bar ``` If the field `threads` is set to `-1` (the default) it will automatically match the number of logical CPUs available. By default almost all Redpanda Connect sources will utilize as many processing threads as have been configured, which makes horizontal scaling easy. --- # Page 337: Resources **URL**: https://docs.redpanda.com/redpanda-connect/configuration/resources.md --- # Resources --- title: Resources latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: resources page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: resources.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/resources.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- Resources are components within Redpanda Connect that are declared with a unique label and can be referenced any number of times within a configuration. Only one instance of each named resource is created, but it is safe to use it in multiple places as they can be shared without consequence. Some components such as caches and rate limits can _only_ be created as a resource. However, for components where it’s optional there are a few reasons why it might be advantageous to do so. ```yaml input: resource: foo pipeline: processors: - resource: bar - cache: operator: set resource: baz key: ${! json("id") } value: ${! content() } output: resource: buz input_resources: - label: foo file: paths: [ ./in.txt ] processor_resources: - label: bar mapping: 'root = content.lowercase()' cache_resources: - label: baz memory: default_ttl: 300s output_resources: - label: buz file: path: ./out.txt ``` ## [](#reusability)Reusability Sometimes it’s necessary to use a rather large component multiple times. Instead of copy/pasting the configuration or using YAML anchors you can define your component as a resource. In the following example we want to make an HTTP request with our payloads. Occasionally the payload might get rejected due to garbage within its contents, and so we catch these rejected requests, attempt to "cleanse" the contents and try to make the same HTTP request again. Since the HTTP request component is quite large (and likely to change over time) we make sure to avoid duplicating it by defining it as a resource `get_foo`: ```yaml pipeline: processors: - resource: get_foo - catch: - mapping: | root = this root.content = this.content.strip_html() - resource: get_foo processor_resources: - label: get_foo http: url: http://example.com/foo verb: POST headers: SomeThing: "set-to-this" SomeThingElse: "set-to-something-else" ``` ## [](#feature-toggling)Feature toggling ### [](#with-environment-variables)With environment variables There are two ways of using resources for feature toggling, the first is to define your feature components with unique names and then apply the old switcheroo with environment variables to select the one you wish to execute: ```yaml pipeline: processors: - resource: ${FEATURE_REQUEST} processor_resources: - label: get_foo http: url: http://example.com/foo verb: POST headers: SomeThing: "set-to-this" SomeThingElse: "set-to-something-else" - label: get_bar http: url: http://example.com/bar verb: PUT headers: Desires: "are-empty" ``` Then when you execute Redpanda Connect use the environment variable to choose your resource: `FEATURE_REQUEST=get_foo rpk connect run ./your_config.yaml`. ### [](#with-imports)With imports However, Redpanda Connect allows you to import resources from separate files with the cli flag `-r` or `-resources`, which can be a useful way to switch out resources with common names based on your chosen environment. For example, with a main configuration file `config.yaml`: ```yaml pipeline: processors: - resource: get_foo ``` And then two resource files, one stored at the path `./staging/request.yaml`: ```yaml processor_resources: - label: get_foo http: url: http://example.com/foo verb: POST headers: SomeThing: "set-to-this" SomeThingElse: "set-to-something-else" ``` And another stored at the path `./production/request.yaml`: ```yaml processor_resources: - label: get_foo http: url: http://example.com/bar verb: PUT headers: Desires: "are-empty" ``` We can select our chosen resource by changing which file we import, either running: ```bash rpk connect run -r ./staging/request.yaml ./config.yaml ``` Or: ```bash rpk connect run -r ./production/request.yaml ./config.yaml ``` These flags also support wildcards, which allows you to import an entire directory of resource files like `rpk connect run -r "./staging/*.yaml" ./config.yaml`. --- # Page 338: Secrets **URL**: https://docs.redpanda.com/redpanda-connect/configuration/secrets.md --- # Secrets --- title: Secrets latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: secrets page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: secrets.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/secrets.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-12-20" --- This topic outlines how to add secrets to a Redpanda Connect configuration without exposing them. ## [](#store-secrets-in-environment-variables)Store secrets in environment variables A common way to securely pass secrets to a service is to use environment variables. Redpanda Connect allows you to inject the values of environment variables into a configuration with the interpolation syntax `${SECRET}`. For example: ```yml thing: super_secret: "${SECRET}" ``` > 📝 **NOTE: Use quotes** > > Use quotes > > It is valid to have `super_secret: ${SECRET}` above (without the quotes), but if `SECRET` is unset then the configuration becomes structurally different. Therefore, it’s always best to wrap environment variable interpolations within quotes so that when the variable is unset you still have a valid configuration (with an empty string). For more information about this syntax, see the [interpolation field page](../interpolation/). ## [](#look-up-secrets-on-a-remote-system-at-runtime)Look up secrets on a remote system at runtime > 📝 **NOTE** > > This feature requires an [enterprise license](../../get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that’s valid for 30 days. Starting with version 4.39.0, you can use the `rpk connect` CLI flag `--secrets` to look up secrets values on a remote system at runtime (for example, in your secrets management solution). This means that Redpanda Connect resolves interpolations in your configuration at runtime without setting environment variables. For example, you could run the following command to retrieve the value for `"${SECRET}"`, when the secret is stored on a Redis server. ```bash rpk connect run ./config.yaml --secrets redis://secret:redis@example.com ``` The command tries to load the secret value from the specified Uniform Resource Name (URN) using the format `scheme://secret:server_address`. You can specify multiple URNs separated by commas, which are tried in turn until a secrets value is successfully returned. ### [](#supported-remote-systems)Supported remote systems You can retrieve secrets from all of the following remote systems. | Remote system | URN format | | --- | --- | | AWS Secret Manager | aws://region/prefixFor example: aws://eu-west-1/redpanda/ | | Azure Key Vault | az://vault-uri/prefix. The vault-uri value should not contain a https:// prefix. | | GCP Secret Manager | gcp://projectID/prefixFor example: gcp://project-id/redpanda- | | Redis | redis://secret:redis@example.com | ## [](#set-secrets-using-shell-commands)Set secrets using shell commands You can set specific fields within a configuration using the CLI flag `--set`, where the syntax is a `=` pair: - ``: A placeholder for the [dot-separated path to the field being set](../field_paths/). - ``: The value you want to set the field to. For example, to write data to a AWS DynamoDB table using the [\`aws\_dynamodb\`output connector](../../components/outputs/aws_dynamodb/), your configuration might look like this: ```yml output: label: "" aws_dynamodb: table: connect-db-test string_columns: id: ${!json("id")} full_content: ${!content()} batching: count: 5 period: 2m credentials: id: ACCESS_KEY_ID secret: SECRET_ACCESS_KEY ``` You could set your credentials to values stored within something like Hashicorp Vault by running the configuration with the `--set` flag and backticks to execute shell commands for the values: ```bash rpk connect run ./config.yaml \ --set "output.aws_dynamodb.credentials.id=`vault kv get -mount=secret access_key_id`" \ --set "output.aws_dynamodb.credentials.secret=`vault kv get -mount=secret secret_access_key`" ``` Using this method lets you inject the secret into the configuration without leaking it into an environment variable. ## [](#avoid-leaked-secrets)Avoid leaked secrets There are a few ways in which configurations parsed by Redpanda Connect can be exported back out of the service. In all of these cases Redpanda Connect will attempt to scrub any field values within the config that are known secrets (any field marked as a secret in the docs). However, if you’re embedding secrets within a configuration outside of the value of secret fields, maybe as part of a Bloblang mapping, then care should be made to avoid exposing the resulting configuration. This specifically means you should not enable [debug HTTP endpoints](../../components/http/about/#debug-endpoints) when the port is exposed, and don’t use the `rpk connect echo` subcommand on configurations containing secrets unless you’re printing to a secure pipe. --- # Page 339: Telemetry **URL**: https://docs.redpanda.com/redpanda-connect/configuration/telemetry.md --- # Telemetry --- title: Telemetry latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: telemetry page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: telemetry.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/telemetry.adoc page-git-created-date: "2024-11-05" page-git-modified-date: "2024-11-19" --- From version 4.38.0, the default Redpanda Connect configuration includes a telemetry service that sends anonymized usage statistics to Redpanda. Redpanda analyzes the data to find out how often each Redpanda Connect component is used in production environments, and common usage patterns. This data analysis aims to help: - Identify gaps in functionality. - Prioritize the delivery of new features, enhancements, and bug fixes. For example, if usage data shows that most `aws_s3` outputs are paired with a `mutation` processor, then embedding a `mutation` field in the component might be a useful enhancement. ## [](#data-collection-method)Data collection method When you execute a configuration, the Redpanda Connect instance sends a JSON payload to the collection server. The payload contains a high-level, anonymized summary of the contents of the configuration file. Field values are never transmitted, nor are decorations of the configuration, such as label names. For example, if you execute this configuration: ```yml input: label: message_input generate: interval: 1s mapping: 'root.input_field = "string"' output: label: message_output aws_s3: bucket: bucket_name path: filename.txt ``` Redpanda extracts the following details from the data collected: - A unique identifier for the Redpanda Connect instance. - How long the configuration has been running for. - Components used in the configuration. In this case, a `generate` input and an `aws_s3` output. - The IP address of the running Redpanda Connect instance. The code responsible for extracting usage data is available in the `connect` repository in the file: [./payload.go](https://github.com/redpanda-data/connect/blob/v4.38.0/internal/telemetry/payload.go). ## [](#data-collection-frequency)Data collection frequency To avoid analyzing instances used for experimentation or testing, a Redpanda Connect instance must have been running for at least 5 minutes before any usage data is collected. Once usage data starts to be emitted, it is sent to the collection server once every 24 hours. ## [](#disable-the-telemetry-service)Disable the telemetry service The telemetry service configuration is only included in the build artifacts released through [GitHub](https://github.com/redpanda-data/connect/releases) or Redpanda’s official [Docker images](https://hub.docker.com/r/redpandadata/connect/). This includes the `rpk connect` plugin, which is used to manage installations and upgrades of Redpanda Connect. Custom builds do not send usage data. To continue to use the official Redpanda artifacts or images but disable the telemetry service, you can either block internet traffic or run the following command: ```bash rpk connect run --disable-telemetry ``` Disabling telemetry does not affect the normal operation of Redpanda Connect. --- # Page 340: Templating **URL**: https://docs.redpanda.com/redpanda-connect/configuration/templating.md --- # Templating --- title: Templating latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: templating page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: templating.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/templating.adoc description: Learn how templates work. page-git-created-date: "2024-05-24" page-git-modified-date: "2025-09-18" --- > ⚠️ **CAUTION** > > Templates are an experimental feature and are subject to change outside major version releases. Templates are a way to define new Redpanda Connect components (similar to plugins) that are implemented by generating a Redpanda Connect configuration snippet from pre-defined parameter fields. This is useful when a common pattern of Redpanda Connect configuration is used but with varying parameters each time. A template is defined in a YAML file that can be imported when Redpanda Connect runs using the flag `-t`: ```bash rpk connect run -t "./templates/*.yaml" ./config.yaml ``` The template describes the type of the component and configuration fields that can be used to customize it, followed by a [Bloblang mapping](../../guides/bloblang/about/) that translates an object containing those fields into a Redpanda Connect configuration structure. This allows you to use logic to generate more complex configurations: #### Template ```yaml name: aws_sqs_list type: input fields: - name: urls type: string kind: list - name: region type: string default: us-east-1 mapping: | root.broker.inputs = this.urls.map_each(url -> { "aws_sqs": { "url": url, "region": this.region, } }) ``` #### Configuration ```yaml input: aws_sqs_list: urls: - https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue1 - https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue2 pipeline: processors: - mapping: | root.id = uuid_v4() root.foo = this.inner.foo root.body = this.outer ``` #### Result ```yaml input: broker: inputs: - aws_sqs: url: https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue1 region: us-east-1 - aws_sqs: url: https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue2 region: us-east-1 pipeline: processors: - mapping: | root.id = uuid_v4() root.foo = this.inner.foo root.body = this.outer ``` You can see more examples of templates on [GitHub](https://github.com/redpanda-data/connect/blob/main/config/template_examples). ## [](#fields)Fields The schema of a template file is as follows: ### [](#name)`name` The name of the component this template will create. **Type**: `string` ### [](#type)`type` The type of the component this template will create. **Type**: `string` Options: `cache` , `input` , `output` , `processor` , `rate_limit` . ### [](#status)`status` The stability of the template describing the likelihood that the configuration spec of the template, or it’s behavior, will change. **Type**: `string` **Default**: `"stable"` | Option | Summary | | --- | --- | | stable | This template is stable and will therefore not change in a breaking way outside of major version releases. | | beta | This template is beta and will therefore not change in a breaking way unless a major problem is found. | | experimental | This template is experimental and therefore subject to breaking changes outside of major version releases. | ### [](#categories)`categories` An optional list of tags, which are used for arbitrarily grouping components in documentation. **Type**: `array` **Default**: `[]` ### [](#summary)`summary` A short summary of the component. **Type**: `string` **Default**: `""` ### [](#description)`description` A longer form description of the component and how to use it. **Type**: `string` **Default**: `""` ### [](#fields-2)`fields` The configuration fields of the template, fields specified here will be parsed from a Redpanda Connect configuration and will be accessible from the template mapping. **Type**: `array` ### [](#fields-name)`fields[].name` The name of the field. **Type**: `string` ### [](#fields-description)`fields[].description` A description of the field. **Type**: `string` **Default**: `""` ### [](#fields-type)`fields[].type` The scalar type of the field. **Type**: `string` | Option | Summary | | --- | --- | | string | standard string type | | string_enum | string type which can have one of a discrete list of values | | string_annotated_enum | string type which can have one of a discrete list of values, where each value must be accompanied by a description that annotates its behaviour in the documentation | | int | standard integer type | | float | standard float type | | bool | A boolean true/false | | bloblang | A bloblang mapping | | unknown | Allows for nesting arbitrary configuration inside a field | ### [](#fields-kind)`fields[].kind` The kind of the field. **Type**: `string` **Default**: `"scalar"` Options: `scalar` , `map` , `list` . ### [](#fields-default)`fields[].default` An optional default value for the field. If a default value is not specified then a configuration without the field is considered incorrect. **Type**: `object` ### [](#fields-advanced)`fields[].advanced` Whether this field is considered advanced. **Type**: `bool` **Default**: `false` ### [](#fields-options)`fields[].options` A list of options for `string_enum` fields, or a map of annotated options for `string_annotated_enum` fields. Required when `fields[].type` is set to `string_enum` or `string_annotated_enum`. - For `string_enum`, provide a non-empty array of string values. - For `string_annotated_enum`, provide a map of `value: "description"` pairs, which are used to annotate behaviour in the docs UI. **Type**: `unknown` ```yml # Examples fields: - name: mode type: string_enum options: ["fast", "balanced", "safe"] - name: delivery type: string_annotated_enum options: at_least_once: "May duplicate messages but never loses them." exactly_once: "Stronger guarantees but with a higher cost and constraints." ``` ### [](#mapping)`mapping` A [Bloblang](../../guides/bloblang/about/) mapping that translates the fields of the template into a valid Redpanda Connect configuration for the target component type. ### [](#metrics_mapping)`metrics_mapping` An optional [Bloblang mapping](../../guides/bloblang/about/) that allows you to rename or prevent certain metrics paths from being exported. For more information check out the [metrics documentation](../../components/metrics/about/#metric-mapping). When metric paths are created, renamed and dropped a trace log is written, enabling TRACE level logging is therefore a good way to diagnose path mappings. Invocations of this mapping are able to reference a metadata field `@label` in order to obtain the value of the label provided to the template configuration. This allows you to match labels with the root of the configuration. **Type**: `string` **Default**: `""` ```yml # Examples metrics_mapping: this.replace("input", "source").replace("output", "sink") metrics_mapping: |- root = if ![ "input_received", "input_latency", "output_sent" ].contains(this) { deleted() } ``` ### [](#tests)`tests` Optional unit test definitions for the template to help you produce valid configurations. The following examples show a test template, associated test configuration, and the commands to execute tests to verify them. **Type**: `array` **Default**: `[]` `test.template.yaml` ```yaml name: basictemplate type: input mapping: | root.generate = { "count": 1, "mapping": "root = \"" + @label.or("") + "\"" } tests: - name: basictemplate test label: quack config: {} expected: generate: count: 1 mapping: root = "quack" ``` `testconfig.yaml` ```yaml input: label: meow basictemplate: {} ``` Test execution: ```bash rpk connect template lint test.template.yaml rpk connect run -t "./test.template.yaml" ./testconfig.yaml ``` ### [](#tests-name)`tests[].name` A name to identify the test. **Type**: `string` ### [](#tests-label)`tests[].label` A label to assign to this template when running the test. **Type**: `string` **Default**: `""` ### [](#tests-config)`tests[].config` A configuration to run this test against. Redpanda Connect checks the results of applying the template with this configuration using a linter. **Type**: `object` ### [](#tests-expected)`tests[].expected` An optional configuration describing the expected result of applying the template, when specified the result will be diffed and any mismatching fields will be reported as a test error. **Type**: `object` --- # Page 341: Unit Testing **URL**: https://docs.redpanda.com/redpanda-connect/configuration/unit_testing.md --- # Unit Testing --- title: Unit Testing latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: unit_testing page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: unit_testing.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/unit_testing.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2026-02-04" --- The Redpanda Connect service offers a command `rpk connect test` for running unit tests on sections of a configuration file. This makes it easy to protect your config files from regressions over time. ## [](#writing-a-test)Writing a test Let’s imagine we have a configuration file `foo.yaml` containing some processors: ```yaml input: kafka: addresses: [ TODO ] topics: [ foo, bar ] consumer_group: foogroup pipeline: processors: - mapping: '"%vend".format(content().uppercase().string())' output: aws_s3: bucket: TODO path: '${! meta("kafka_topic") }/${! json("message.id") }.json' ``` One way to write our unit tests for this config is to accompany it with a file of the same name and extension but suffixed with `_benthos_test`, which in this case would be `foo_benthos_test.yaml`. ```yml tests: - name: example test target_processors: '/pipeline/processors' environment: {} input_batch: - content: 'example content' metadata: example_key: example metadata value output_batches: - - content_equals: EXAMPLE CONTENTend metadata_equals: example_key: example metadata value ``` Under `tests` we have a list of any number of unit tests to execute for the config file. Each test is run in complete isolation, including any resources defined by the config file. Tests should be allocated a unique `name` that identifies the feature being tested. The field `target_processors` is either the label of a processor to test, or a [JSON Pointer](https://tools.ietf.org/html/rfc6901) that identifies the position of a processor, or list of processors, within the file which should be executed by the test. For example a value of `foo` would target a processor with the label `foo`, and a value of `/input/processors` would target all processors within the input section of the config. The field `environment` allows you to define an object of key/value pairs that set environment variables to be evaluated during the parsing of the target config file. These are unique to each test, allowing you to test different environment variable interpolation combinations. The field `input_batch` lists one or more messages to be fed into the targeted processors as a batch. Each message of the batch may have its raw content defined as well as metadata key/value pairs. For the common case where the messages are in JSON format, you can use `json_content` instead of `content` to specify the message structurally rather than verbatim. The field `output_batches` lists any number of batches of messages which are expected to result from the target processors. Each batch lists any number of messages, each one defining [`conditions`](#output-conditions) to describe the expected contents of the message. If the number of batches defined does not match the resulting number of batches the test will fail. If the number of messages defined in each batch does not match the number in the resulting batches the test will fail. If any condition of a message fails then the test fails. ### [](#inline-tests)Inline tests Sometimes it’s more convenient to define your tests within the config being tested. This is fine, simply add the `tests` field to the end of the config being tested. ### [](#bloblang-tests)Bloblang tests Sometimes when working with large [Bloblang mappings](../../guides/bloblang/about/) it’s preferred to have the full mapping in a separate file to your Redpanda Connect configuration. In this case it’s possible to write unit tests that target and execute the mapping directly with the field `target_mapping`, which when specified is interpreted as either an absolute path or a path relative to the test definition file that points to a file containing only a Bloblang mapping. For example, if we were to have a file `cities.blobl` containing a mapping: ```bloblang root.Cities = this.locations. filter(loc -> loc.state == "WA"). map_each(loc -> loc.name). sort().join(", ") ``` We can accompany it with a test file `cities_test.yaml` containing a regular test definition: ```yml tests: - name: test cities mapping target_mapping: './cities.blobl' environment: {} input_batch: - content: | { "locations": [ {"name": "Seattle", "state": "WA"}, {"name": "New York", "state": "NY"}, {"name": "Bellevue", "state": "WA"}, {"name": "Olympia", "state": "WA"} ] } output_batches: - - json_equals: {"Cities": "Bellevue, Olympia, Seattle"} ``` And execute this test the same way we execute other Redpanda Connect tests (`rpk connect test ./dir/cities_test.yaml`, `rpk connect test ./dir/…​`, etc). ### [](#fragmented-tests)Fragmented tests Sometimes the number of tests you need to define in order to cover a config file is so vast that it’s necessary to split them across multiple test definition files. This is possible but Redpanda Connect still requires a way to detect the configuration file being targeted by these fragmented test definition files. In order to do this we must prefix our `target_processors` field with the path of the target relative to the definition file. The syntax of `target_processors` in this case is a full [JSON Pointer](https://tools.ietf.org/html/rfc6901) that should look something like `target.yaml#/pipeline/processors`. For example, if we saved our test definition above in an arbitrary location like `./tests/first.yaml` and wanted to target our original `foo.yaml` config file, we could do that with the following: ```yml tests: - name: example test target_processors: '../foo.yaml#/pipeline/processors' environment: {} input_batch: - content: 'example content' metadata: example_key: example metadata value output_batches: - - content_equals: EXAMPLE CONTENTend metadata_equals: example_key: example metadata value ``` ## [](#input-definitions)Input Definitions ### [](#content)`content` Sets the raw content of the message. ### [](#json_content)`json_content` ```yml json_content: foo: foo value bar: [ element1, 10 ] ``` Sets the raw content of the message to a JSON document matching the structure of the value. ### [](#file_content)`file_content` ```yml file_content: ./foo/bar.txt ``` Sets the raw content of the message by reading a file. The path of the file should be relative to the path of the test file. ### [](#metadata)`metadata` A map of key/value pairs that sets the metadata values of the message. ## [](#output-conditions)Output Conditions ### [](#bloblang)`bloblang` ```yml bloblang: 'this.age > 10 && @foo.length() > 0' ``` Executes a [Bloblang expression](../../guides/bloblang/about/) on a message, if the result is anything other than a boolean equalling `true` the test fails. ### [](#content_equals)`content_equals` ```yml content_equals: example content ``` Checks the full raw contents of a message against a value. ### [](#content_matches)`content_matches` ```yml content_matches: "^foo [a-z]+ bar$" ``` Checks whether the full raw contents of a message matches a regular expression (re2). ### [](#metadata_equals)`metadata_equals` ```yml metadata_equals: example_key: example metadata value ``` Checks a map of metadata keys to values against the metadata stored in the message. If there is a value mismatch between a key of the condition versus the message metadata this condition will fail. ### [](#file_equals)`file_equals` ```yml file_equals: ./foo/bar.txt ``` Checks that the contents of a message matches the contents of a file. The path of the file should be relative to the path of the test file. ### [](#file_json_equals)`file_json_equals` ```yml file_json_equals: ./foo/bar.json ``` Checks that both the message and the file contents are valid JSON documents, and that they are structurally equivalent. Will ignore formatting and ordering differences. The path of the file should be relative to the path of the test file. ### [](#json_equals)`json_equals` ```yml json_equals: { "key": "value" } ``` Checks that both the message and the condition are valid JSON documents, and that they are structurally equivalent. Will ignore formatting and ordering differences. You can also structure the condition content as YAML and it will be converted to the equivalent JSON document for testing: ```yml json_equals: key: value ``` ### [](#json_contains)`json_contains` ```yml json_contains: { "key": "value" } ``` Checks that both the message and the condition are valid JSON documents, and that the message is a superset of the condition. ## [](#running-tests)Running tests Executing tests for a specific config can be done by pointing the subcommand `test` at either the config to be tested or its test definition, e.g. `rpk connect test ./config.yaml` and `rpk connect test ./config_benthos_test.yaml` are equivalent. The `test` subcommand also supports wildcard patterns e.g. `rpk connect test ./foo/*.yaml` will execute all tests within matching files. In order to walk a directory tree and execute all tests found you can use the shortcut `./…​`, e.g. `rpk connect test ./…​` will execute all tests found in the current directory, any child directories, and so on. If you want to allow components to write logs at a provided level to stdout when running the tests, you can use `rpk connect test --log `. Please consult the [logger docs](../../components/logger/about/) for further details. ## [](#mocking-processors)Mocking processors BETA: This feature is currently in a BETA phase, which means breaking changes could be made if a fundamental issue with the feature is found. Sometimes you’ll want to write tests for a series of processors, where one or more of them are networked (or otherwise stateful). Rather than creating and managing mocked services you can define mock versions of those processors in the test definition. For example, if we have a config with the following processors: ```yaml pipeline: processors: - mapping: 'root = "simon says: " + content()' - label: get_foobar_api http: url: http://example.com/foobar verb: GET - mapping: 'root = content().uppercase()' ``` Rather than create a fake service for the `http` processor to interact with we can define a mock in our test definition that replaces it with a [`mapping` processor](../../components/processors/mapping/). Mocks are configured as a map of labels that identify a processor to replace and the config to replace it with: ```yaml tests: - name: mocks the http proc target_processors: '/pipeline/processors' mocks: get_foobar_api: mapping: 'root = content().string() + " this is some mock content"' input_batch: - content: "hello world" output_batches: - - content_equals: "SIMON SAYS: HELLO WORLD THIS IS SOME MOCK CONTENT" ``` With the above test definition the `http` processor will be swapped out for `mapping: 'root = content().string() + " this is some mock content"'`. For the purposes of mocking it is recommended that you use a [`mapping` processor](../../components/processors/mapping/) that simply mutates the message in a way that you would expect the mocked processor to. > 📝 **NOTE** > > It’s not currently possible to mock components that are imported as separate resource files (using `--resource`/`-r`). It is recommended that you mock these by maintaining separate definitions for test purposes (`-r "./test/*.yaml"`). ### [](#more-granular-mocking)More granular mocking It is also possible to target specific fields within the test config by [JSON pointers](https://tools.ietf.org/html/rfc6901) as an alternative to labels. The following test definition would create the same mock as the previous: ```yaml tests: - name: mocks the http proc target_processors: '/pipeline/processors' mocks: /pipeline/processors/1: mapping: 'root = content().string() + " this is some mock content"' input_batch: - content: "hello world" output_batches: - - content_equals: "SIMON SAYS: HELLO WORLD THIS IS SOME MOCK CONTENT" ``` ## [](#fields)Fields The schema of a template file is as follows: ### [](#tests)`tests` A list of one or more unit tests to execute. **Type**: `array` ### [](#tests-name)`tests[].name` The name of the test, this should be unique and give a rough indication of what behavior is being tested. **Type**: `string` ### [](#tests-environment)`tests[].environment` An optional map of environment variables to set for the duration of the test. **Type**: `object` ### [](#tests-target_processors)`tests[].target_processors` A \[JSON Pointer\]\[json-pointer\] that identifies the specific processors which should be executed by the test. The target can either be a single processor or an array of processors. Alternatively a resource label can be used to identify a processor. It is also possible to target processors in a separate file by prefixing the target with a path relative to the test file followed by a # symbol. **Type**: `string` **Default**: `"/pipeline/processors"` ```yml # Examples target_processors: foo_processor target_processors: /pipeline/processors/0 target_processors: target.yaml#/pipeline/processors target_processors: target.yaml#/pipeline/processors ``` ### [](#tests-target_mapping)`tests[].target_mapping` A file path relative to the test definition path of a Bloblang file to execute as an alternative to testing processors with the `target_processors` field. This allows you to define unit tests for Bloblang mappings directly. **Type**: `string` **Default**: `""` ### [](#tests-mocks)`tests[].mocks` An optional map of processors to mock. Keys should contain either a label or a JSON pointer of a processor that should be mocked. Values should contain a processor definition, which will replace the mocked processor. Most of the time you’ll want to use a \[`mapping` processor\]\[processors.mapping\] here, and use it to create a result that emulates the target processor. **Type**: `object` ```yml # Examples mocks: get_foobar_api: mapping: root = content().string() + " this is some mock content" mocks: /pipeline/processors/1: mapping: root = content().string() + " this is some mock content" ``` ### [](#tests-input_batch)`tests[].input_batch` Define a batch of messages to feed into your test, specify either an `input_batch` or a series of `input_batches`. **Type**: `array` ### [](#tests-input_batch-content)`tests[].input_batch[].content` The raw content of the input message. **Type**: `string` ### [](#tests-input_batch-json_content)`tests[].input_batch[].json_content` Sets the raw content of the message to a JSON document matching the structure of the value. **Type**: `object` ```yml # Examples json_content: bar: - element1 - 10 foo: foo value ``` ### [](#tests-input_batch-file_content)`tests[].input_batch[].file_content` Sets the raw content of the message by reading a file. The path of the file should be relative to the path of the test file. **Type**: `string` ```yml # Examples file_content: ./foo/bar.txt ``` ### [](#tests-input_batch-metadata)`tests[].input_batch[].metadata` A map of metadata key/values to add to the input message. **Type**: `object` ### [](#tests-input_batches)`tests[].input_batches` Define a series of batches of messages to feed into your test, specify either an `input_batch` or a series of `input_batches`. **Type**: `two-dimensional array` ### [](#tests-input_batches-content)`tests[].input_batches[][].content` The raw content of the input message. **Type**: `string` ### [](#tests-input_batches-json_content)`tests[].input_batches[][].json_content` Sets the raw content of the message to a JSON document matching the structure of the value. **Type**: `object` ```yml # Examples json_content: bar: - element1 - 10 foo: foo value ``` ### [](#tests-input_batches-file_content)`tests[].input_batches[][].file_content` Sets the raw content of the message by reading a file. The path of the file should be relative to the path of the test file. **Type**: `string` ```yml # Examples file_content: ./foo/bar.txt ``` ### [](#tests-input_batches-metadata)`tests[].input_batches[][].metadata` A map of metadata key/values to add to the input message. **Type**: `object` ### [](#tests-output_batches)`tests[].output_batches` List of output batches. **Type**: `two-dimensional array` ### [](#tests-output_batches-bloblang)`tests[].output_batches[][].bloblang` Executes a Bloblang mapping on the output message, if the result is anything other than a boolean equalling `true` the test fails. **Type**: `string` ```yml # Examples bloblang: this.age > 10 && @foo.length() > 0 ``` ### [](#tests-output_batches-content_equals)`tests[].output_batches[][].content_equals` Checks the full raw contents of a message against a value. **Type**: `string` ### [](#tests-output_batches-content_matches)`tests[].output_batches[][].content_matches` Checks whether the full raw contents of a message matches a regular expression (re2). **Type**: `string` ```yml # Examples content_matches: ^foo [a-z]+ bar$ ``` ### [](#tests-output_batches-metadata_equals)`tests[].output_batches[][].metadata_equals` Checks a map of metadata keys to values against the metadata stored in the message. If there is a value mismatch between a key of the condition versus the message metadata this condition will fail. **Type**: `object` ```yml # Examples metadata_equals: example_key: example metadata value ``` ### [](#tests-output_batches-file_equals)`tests[].output_batches[][].file_equals` Checks that the contents of a message matches the contents of a file. The path of the file should be relative to the path of the test file. **Type**: `string` ```yml # Examples file_equals: ./foo/bar.txt ``` ### [](#tests-output_batches-file_json_equals)`tests[].output_batches[][].file_json_equals` Checks that both the message and the file contents are valid JSON documents, and that they are structurally equivalent. Will ignore formatting and ordering differences. The path of the file should be relative to the path of the test file. **Type**: `string` ```yml # Examples file_json_equals: ./foo/bar.json ``` ### [](#tests-output_batches-json_equals)`tests[].output_batches[][].json_equals` Checks that both the message and the condition are valid JSON documents, and that they are structurally equivalent. Will ignore formatting and ordering differences. **Type**: `object` ```yml # Examples json_equals: key: value ``` ### [](#tests-output_batches-json_contains)`tests[].output_batches[][].json_contains` Checks that both the message and the condition are valid JSON documents, and that the message is a superset of the condition. **Type**: `object` ```yml # Examples json_contains: key: value ``` ### [](#tests-output_batches-file_json_contains)`tests[].output_batches[][].file_json_contains` Checks that both the message and the file contents are valid JSON documents, and that the message is a superset of the condition. Will ignore formatting and ordering differences. The path of the file should be relative to the path of the test file. **Type**: `string` ```yml # Examples file_json_contains: ./foo/bar.json ``` --- # Page 342: Using CUE **URL**: https://docs.redpanda.com/redpanda-connect/configuration/using_cue.md --- # Using CUE --- title: Using CUE latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: using_cue page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: using_cue.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/using_cue.adoc page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- > ⚠️ **CAUTION** > > CUE support is experimental. It may change for some time to improve CUE’s ability to type-check Redpanda Connect configurations at the expense of causing new validation errors when moving from one Redpanda Connect release to the next. [**CUE**](https://cuelang.org/) is a powerful configuration language that makes it easier and safer to build Redpanda Connect configurations. It achieves this by validating and type-checking configurations as well as allowing you to build useful utilities that reduce boilerplate. In this guide, we will see how to build a Redpanda Connect configuration using CUE, export it to YAML and execute it. ## [](#prerequisites)Prerequisites Before you get started, ensure that you have installed CUE by [following this guide](https://cuelang.org/docs/install/). If this is your first time workin with it, then it’s a great idea to step through the [CUE tutorial](https://cuelang.org/docs/tutorial/) and familiarize yourself with the language. ## [](#create)Create Create a directory for the CUE module that will contain the configuration: ```shell mkdir hello-cue cd hello-cue cue mod init example.com/hello-cue touch config.cue ``` > CUE modules must start with a hostname. This will typically be the URL of your repository. For example: `cue mod init github.com/redpanda-data/hello-cue`. The `rpk connect list` command will generate a CUE package containing the types we’ll need to build our configuration. Let’s write this package into our project: ```bash mkdir benthos rpk connect list --format cue > benthos/schema.cue ``` At this point, you should now have the following directory structure: hello-cue/ benthos/ schema.cue cue.mod/ pkg/ usr/ module.cue config.cue We are now ready to write our Redpanda Connect config in CUE. Let’s start by editing our `config.cue` to include the following snippet: ```cue import "example.com/hello-cue/benthos" benthos.#Config & { input: { generate: { mapping: """ root = { "message": "Hello, CUE!" } """ } } pipeline: { processors: [ { mapping: """ root = this root.id = uuid_v4() """ } ] } output: { stdout: {} } } ``` Let’s see what this will look like as YAML by running `cue export` while in the `hello-cue` directory: ```shell cue export --out yaml config.cue ``` This should output something like this: ```yaml input: generate: mapping: 'root = { "message": "Hello, CUE!" }' pipeline: processors: - mapping: |- root = this root.id = uuid_v4() output: stdout: {} tests: [] ``` We can run this with Redpanda Connect to see that it indeed works: ```bash rpk connect run <(cue export --out yaml config.cue) ``` When you are satisfied with the results, terminate the Redpanda Connect process and let’s move on to look at some of the nice features that we get with CUE. ## [](#enhance)Enhance The `config.cue` above looks eerily like JSON. This is because CUE is a superset of JSON and shares its syntax. However, we can shorten our configuration to reduce indentation and curly brackets. Let’s rewrite `config.cue` to look like this: ```cue import "example.com/hello-cue/benthos" benthos.#Config & { input: generate: mapping: """ root = { "message": "Hello, CUE!" } """ pipeline: processors: [ { mapping: """ root = this root.id = uuid_v4() """ } ] output: stdout: {} } ``` If you run the same `cue export` command from earlier, you’ll notice that the YAML output is the same. Next, we’ll look at what some error handling patterns might look like with CUE. One typical technique to detect messages with errors is to use the `switch` output to wrap another output with some error detection and handling. Another pattern involves limiting the number of retries on a given output that is misbehaving and rejecting or dropping messages with some useful logging. If we combine all these concepts together we get: ```yaml output: switch: cases: - check: errored() output: reject: "failed to process message: ${! error() }" - output: retry: max_retries: 5 output: gcp_pubsub: project: "sample-project" topic: "sample-topic" ``` There are quite a few lines of YAML here and we seem to be going sideways as we compose more functionality. We can try and make this more manageable with CUE! Let’s create a new file in our `hello-cue` directory called `benthos/helpers.cue`: ```shell touch benthos/helpers.cue ``` In this file, add the following snippet: ```cue package benthos #Guarded: self = { // The desired output that will be wrapped with error handling mechanisms #output: #Output // The error text to emit if the output receives any messages which contained // processing errors #errorMessage: string // The number of retries to attempt on the desired output (default is 3) #maxRetries: uint | *3 // The error message to emit if the retry attempts are exhausted #retryErrorMessage: string // Whether to drop or reject any failed messages #errorHandling: "drop" | "reject" switch: cases: [ { check: "errored()" output: { if self.#errorHandling == "reject" { reject: self.#errorMessage } if self.#errorHandling == "drop" { drop: {} processors: [{ log: message: self.#errorMessage }] } } }, { output: fallback: [ { retry: { max_retries: self.#maxRetries output: self.#output } }, { if self.#errorHandling == "reject" { reject: self.#retryErrorMessage } if self.#errorHandling == "drop" { drop: {} processors: [{ log: message: self.#retryErrorMessage }] } } ] } ] } ``` Now, let’s get back to `config.cue` and edit a few bits while leveraging this helper: ```cue import "example.com/hello-cue/benthos" benthos.#Config & { input: generate: { count: 1 interval: "0" mapping: """ root = { "message": "Hello, CUE!" } """ } output: benthos.#Guarded & { #errorMessage: "failed to process message: ${! error() }" #maxRetries: 3 #retryErrorMessage: "failed to output message after \(#maxRetries) retries" #errorHandling: "drop" #output: http_client: { url: "http://localhost:4195/sad-blob" retries: 0 } } } ``` If you rerun `cue export` now, you’ll see that we’ve wrapped our output with a couple of error handling mechanisms. We also had access to powerful CUE features like conditional fields based on `#errorHandling`, default values and interpolations. ```yaml input: generate: count: 1 interval: "0" mapping: 'root = { "message": "Hello, CUE!" }' output: switch: cases: - check: errored() output: drop: {} processors: - log: message: 'failed to process message: ${! error() }' - output: fallback: - retry: max_retries: 3 output: http_client: url: http://localhost:4195/sad-blob retries: 0 - drop: {} processors: - log: message: failed to output message after 3 retries tests: [] ``` The final directory structure of your hello-cue project should look like this: hello-cue/ benthos/ schema.cue helpers.cue cue.mod/ pkg/ usr/ module.cue config.cue ## [](#included-cue-types)Included CUE types The `benthos.cue` file we emitted earlier contains a number of useful types that we can use when build configuration files and helpers. These include: - `benthos.#Config` This definition describes the format of a Redpanda Connect config file. You’ll want to use it at the top of your configuration file to validate its overall structure. - `benthos.#Input` - `benthos.#Output` - `benthos.#Processor` - `benthos.#RateLimit` - `benthos.#Buffer` - `benthos.#Cache` - `benthos.#Metric` - `benthos.#Tracer` Each of these definitions is a disjunction that holds all the corresponding components in Redpanda Connect. In other words, a CUE field that is specified as `benthos.#Input`, such as `myfield: benthos.#Input`, must resolve to a valid Redpanda Connect input. ## [](#wrap-up)Wrap up Being able to define helper packages and definitions like `#Guarded` and reusing them across your Redpanda Connect configurations is a really powerful feature of CUE. This will allow you to share consistent good practices without messy boilerplate across projects and teams! --- # Page 343: Windowed Processing **URL**: https://docs.redpanda.com/redpanda-connect/configuration/windowed_processing.md --- # Windowed Processing --- title: Windowed Processing latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: windowed_processing page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: windowed_processing.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/configuration/pages/windowed_processing.adoc description: Learn how to process periodic windows of messages with Redpanda Connect. page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- A window is a batch of messages made with respect to time, with which we are able to perform processing that can analyze or aggregate the messages of the window. This is useful in stream processing as the dataset is never "complete", and therefore in order to perform analysis against a collection of messages we must do so by creating a continuous feed of windows (collections), where our analysis is made against each window. For example, given a stream of messages relating to cars passing through various traffic lights: ```json { "traffic_light": "cbf2eafc-806e-4067-9211-97be7e42cee3", "created_at": "2021-08-07T09:49:35Z", "registration_plate": "AB1C DEF", "passengers": 3 } ``` Windowing allows us to produce a stream of messages representing the total traffic for each light every hour: ```json { "traffic_light": "cbf2eafc-806e-4067-9211-97be7e42cee3", "created_at": "2021-08-07T10:00:00Z", "unique_cars": 15, "passengers": 43 } ``` ## [](#creating-windows)Creating windows The first step in processing windows is producing the windows themselves, this can be done by configuring a window producing buffer after your input: ### System A `system_window` buffer creates windows by following the system clock of the running machine. Windows will be created and emitted at predictable times, but this also means windows for historic data will not be emitted and therefore prevents backfills of traffic data: ```yaml input: kafka: addresses: [ TODO ] topics: [ traffic_data ] consumer_group: traffic_consumer checkpoint_limit: 1000 buffer: system_window: timestamp_mapping: root = this.created_at size: 1h allowed_lateness: 3m ``` For more information about this buffer refer to the `system_window` buffer docs. ## [](#grouping)Grouping With a window buffer chosen our stream of messages will be emitted periodically as batches of all messages that fit within each window. Since we want to analyse the window separately for each traffic light we need to expand this single batch out into one for each traffic light identifier within the window. For that purpose we have two processor options: [`group_by`](../../components/processors/group_by/) and [`group_by_value`](../../components/processors/group_by_value/). In our case we want to group by the value of the field `traffic_light` of each message, which we can do with the following: ```yaml pipeline: processors: - group_by_value: value: ${! json("traffic_light") } ``` ## [](#aggregating)Aggregating Once our window has been grouped the next step is to calculate the aggregated passenger and unique cars counts. For this purpose the Redpanda Connect [mapping language Bloblang](../../guides/bloblang/about/) comes in handy as the method [`from_all`](../../guides/bloblang/methods/#from_all) executes the target function against the entire batch and returns an array of the values, allowing us to mutate the result with chained methods such as [`sum`](../../guides/bloblang/methods/#sum): ```yaml pipeline: processors: - group_by_value: value: ${! json("traffic_light") } - mapping: | let is_first_message = batch_index() == 0 root.traffic_light = this.traffic_light root.created_at = @window_end_timestamp root.total_cars = if $is_first_message { json("registration_plate").from_all().unique().length() } root.passengers = if $is_first_message { json("passengers").from_all().sum() } # Only keep the first batch message containing the aggregated results. root = if ! $is_first_message { deleted() } ``` [Bloblang](../../guides/bloblang/about/) is very powerful, and by using [`from`](../../guides/bloblang/methods/#from) and [`from_all`](../../guides/bloblang/methods/#from_all) it’s possible to perform a wide range of batch-wide processing. If you fancy a challenge try updating the above mapping to only count passengers from the first journey of each registration plate in the window (hint: the [`fold` method](../../guides/bloblang/methods/#fold) might come in handy). --- # Page 344: Cookbooks **URL**: https://docs.redpanda.com/redpanda-connect/cookbooks.md --- # Cookbooks --- title: Cookbooks latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: index page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: index.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/index.adoc page-git-created-date: "2024-06-03" page-git-modified-date: "2024-06-04" --- - [DynamoDB CDC Patterns](dynamodb_cdc/) Learn how to capture, filter, transform, and route DynamoDB change data capture (CDC) events with Redpanda Connect. - [Oracle Database CDC Patterns](oracledb_cdc/) Learn how to capture, filter, transform, and route Oracle Database change data capture (CDC) events with Redpanda Connect. - [Enrichment Workflows](enrichments/) How to configure Redpanda Connect to process a workflow of enrichment services. - [Filtering and Sampling](filtering/) Configure Redpanda Connect to conditionally drop messages. - [Work with Jira Issues](jira/) Learn how to use the Jira processor as both an input and output in Redpanda Connect. - [Joining Streams](joining_streams/) How to hydrate documents by joining multiple streams. - [Ingest data into Snowflake](snowflake_ingestion/) Configure Redpanda Connect to ingest data from a Redpanda topic into Snowflake using Snowpipe Streaming. - [Redpanda Migrator](redpanda_migrator/) Move your workloads from any Kafka system to Redpanda using a single command. Redpanda Migrator lets you migrate Kafka messages, schemas, and ACLs quickly and efficiently. - [Retrieval-Augmented Generation (RAG)](rag/) How to configure Redpanda Connect to create a RAG pipeline, using PostgreSQL and PGVector. - [Custom Metrics](custom_metrics/) Learn how to emit custom metrics from messages. - [Create a Discord Bot](discord_bot/) Learn how to use Redpanda Connect to create a Discord bot. --- # Page 345: Custom Metrics **URL**: https://docs.redpanda.com/redpanda-connect/cookbooks/custom_metrics.md --- # Custom Metrics --- title: Custom Metrics latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: custom_metrics page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: custom_metrics.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/custom_metrics.adoc description: Learn how to emit custom metrics from messages. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-03-31" --- You can’t build graphs without metrics, and [Redpanda Connect emits many](../../components/metrics/about/). However, occasionally you might want to also emit custom metrics that track data extracted from messages being processed. In this cookbook we’ll explore how to achieve this by configuring Redpanda Connect to pull download stats from Github, DockerHub and Homebrew and emit them as gauges. ## [](#the-basics)The basics Firstly, we need to target an API so let’s start with the nice and simple Homebrew API, which we’ll poll every 60 seconds. We can either do it with an [`http_client` input](../../components/inputs/http_client/) and a [rate limit](../../components/rate_limits/about/) that restricts us to one request per 60 seconds, or we can use a [`generate` input](../../components/inputs/generate/) to generate a message every 60 seconds that triggers an [`http` processor](../../components/processors/http/): ### Processor ```yaml input: generate: interval: 60s mapping: root = "" pipeline: processors: - http: url: https://formulae.brew.sh/api/formula/benthos.json verb: GET ``` ### Input ```yaml input: http_client: url: https://formulae.brew.sh/api/formula/benthos.json verb: GET rate_limit: brewlimit rate_limit_resources: - label: brewlimit local: count: 1 interval: 60s ``` For this cookbook we’ll continue with the processor option as it makes it easier to deploy it as a [scheduled lambda function](../../guides/serverless/lambda/) later on, which is how I’m currently doing it in real life. The homebrew formula API gives us a JSON blob that looks like this (removing fields we’re not interested in, and with numbers inflated relative to my ego): ```json { "name":"benthos", "desc":"Stream processor for mundane tasks written in Go", "analytics":{"install":{"30d":{"benthos":78978979},"90d":{"benthos":253339124},"365d":{"benthos":681356871}}} } ``` This format makes it fairly easy to emit the value of `analytics.install.30d.benthos` as a gauge with the [`metric` processor](../../components/processors/metric/): ```yaml http: address: 0.0.0.0:4195 input: generate: interval: 60s mapping: root = "" pipeline: processors: - http: url: https://formulae.brew.sh/api/formula/benthos.json verb: GET - metric: type: gauge name: downloads labels: source: homebrew value: ${! json("analytics.install.30d.benthos") } - mapping: root = deleted() metrics: mapping: if this != "downloads" { deleted() } prometheus: {} ``` With the above config we have selected the [`prometheus` metrics type](../../components/metrics/prometheus/), which allows us to use [Prometheus](https://prometheus.io/) to scrape metrics from Redpanda Connect by polling its HTTP API at the url `http://localhost:4195/stats`. We have also specified a \[`path_mapping`\]\[metrics.prometheus.path\_mapping\] that deletes any internal metrics usually emitted by Redpanda Connect by filtering on our custom metric name. Finally, there’s also a [`mapping` processor](../../components/processors/mapping/) added to the end of our pipeline that deletes all messages since we’re not interested in sending the raw data anywhere after this point anyway. While running this config you can verify that our custom metric is emitted with `curl`: ```sh curl -s http://localhost:4195/stats | grep downloads ``` Giving something like: ```text # HELP benthos_downloads Redpanda Connect Gauge metric # TYPE benthos_downloads gauge benthos_downloads{source="homebrew"} 78978979 ``` Easy! The DockerHub API is also pretty simple, and adding it to our pipeline is just: ### Diff ```diff source: homebrew value: ${! json("analytics.install.30d.benthos") } + - mapping: root = "" + + - http: + url: http://docker.redpanda.com/redpandadata/connect + verb: GET + headers: + Content-Type: application/json + + - metric: + type: gauge + name: downloads + labels: + source: dockerhub + value: ${! json("pull_count") } + - mapping: root = deleted() ``` ### Full Config ```yaml http: address: 0.0.0.0:4195 input: generate: interval: 60s mapping: root = "" pipeline: processors: - http: url: https://formulae.brew.sh/api/formula/benthos.json verb: GET - metric: type: gauge name: downloads labels: source: homebrew value: ${! json("analytics.install.30d.benthos") } - mapping: root = "" - http: url: http://docker.redpanda.com/redpandadata/connect verb: GET headers: Content-Type: application/json - metric: type: gauge name: downloads labels: source: dockerhub value: ${! json("pull_count") } - mapping: root = deleted() metrics: mapping: if this != "downloads" { deleted() } prometheus: {} ``` ## [](#advanced-custom-metrics-example)Advanced custom metrics example So that’s the basics covered. Next, we’re going to target the Github releases API which gives a slightly more complex payload that looks something like this: ```json [ { "tag_name": "X.XX.X", "assets":[ {"name":"benthos-lambda_X.XX.X_linux_amd64.zip","download_count":543534545}, {"name":"benthos_X.XX.X_darwin_amd64.tar.gz","download_count":43242342}, {"name":"benthos_X.XX.X_freebsd_amd64.tar.gz","download_count":534565656}, {"name":"benthos_X.XX.X_linux_amd64.tar.gz","download_count":743282474324} ] } ] ``` It’s an array of objects, one for each tagged release, with a field `assets` which is an array of objects representing each release asset, of which we want to emit a separate download gauge. In order to do this we’re going to use a [`mapping` processor](../../components/processors/mapping/) to remap the payload from Github into an array of objects of the following form: ```json [ {"source":"github","dist":"lambda_linux_amd64","download_count":543534545,"version":"X.XX.X"}, {"source":"github","dist":"darwin_amd64","download_count":43242342,"version":"X.XX.X"}, {"source":"github","dist":"freebsd_amd64","download_count":534565656,"version":"X.XX.X"}, {"source":"github","dist":"linux_amd64","download_count":743282474324,"version":"X.XX.X"} ] ``` Then we can use an [`unarchive` processor](../../components/processors/unarchive/) with the format `json_array` to expand this array into N individual messages, one for each asset. Finally, we will follow up with a [`metric` processor](../../components/processors/metric/) that dynamically sets labels following the fields `source`, `dist` and `version` so that we have a separate metrics series for each asset type for each tagged version. A simple pipeline of these steps would look like this (please forgive the regexp): ```yaml http: address: 0.0.0.0:4195 input: generate: interval: 60s mapping: root = "" pipeline: processors: - http: url: https://api.github.com/repos/redpanda-data/connect/releases verb: GET - mapping: | root = this.map_each(release -> release.assets.map_each(asset -> { "source": "github", "dist": asset.name.re_replace_all("^benthos-?((lambda_)|_)[0-9\\.]+(-rc[0-9]+)?_([^\\.]+).*", "$2$4"), "download_count": asset.download_count, "version": release.tag_name.trim("v"), }).filter(asset -> asset.dist != "checksums")).flatten() - unarchive: format: json_array - metric: type: gauge name: downloads labels: dist: ${! json("dist") } source: ${! json("source") } value: ${! json("download_count") } - mapping: root = deleted() metrics: mapping: if this != "downloads" { deleted() } prometheus: {} ``` Finally, let’s combine all the custom metrics into one pipeline. ## [](#combining-into-a-workflow)Combining into a workflow The following config expands on the previous examples by configuring each API poll as a [`branch` processor](../../components/processors/branch/), which allows us to run them within a [`workflow` processor](../../components/processors/workflow/) that can execute all three branches in parallel. The [`metric` processors](../../components/processors/metric/) have also been combined into a single reusable resource by updating the other API calls to format their payloads into the same structure as our Github remap. ```yaml http: address: 0.0.0.0:4195 input: generate: interval: 60s mapping: root = {} pipeline: processors: - workflow: meta_path: results order: [ [ dockerhub, github, homebrew ] ] processor_resources: - label: dockerhub branch: request_map: 'root = ""' processors: - try: - http: url: http://docker.redpanda.com/redpandadata/connect verb: GET headers: Content-Type: application/json - mapping: | root.source = "docker" root.dist = "docker" root.download_count = this.pull_count root.version = "all" - resource: metric_gauge - label: github branch: request_map: 'root = ""' processors: - try: - http: url: https://api.github.com/repos/redpanda-data/connect/releases verb: GET - mapping: | root = this.map_each(release -> release.assets.map_each(asset -> { "source": "github", "dist": asset.name.re_replace_all("^benthos-?((lambda_)|_)[0-9\\.]+(-rc[0-9]+)?_([^\\.]+).*", "$2$4"), "download_count": asset.download_count, "version": release.tag_name.trim("v"), }).filter(asset -> asset.dist != "checksums")).flatten() - unarchive: format: json_array - resource: metric_gauge - mapping: 'root = if batch_index() != 0 { deleted() }' - label: homebrew branch: request_map: 'root = ""' processors: - try: - http: url: https://formulae.brew.sh/api/formula/benthos.json verb: GET - mapping: | root.source = "homebrew" root.dist = "homebrew" root.download_count = this.analytics.install.30d.benthos root.version = "all" - resource: metric_gauge - label: metric_gauge metric: type: gauge name: downloads labels: dist: ${! json("dist") } source: ${! json("source") } version: ${! json("version") } value: ${! json("download_count") } metrics: mapping: if this != "downloads" { deleted() } prometheus: {} ``` --- # Page 346: Create a Discord Bot **URL**: https://docs.redpanda.com/redpanda-connect/cookbooks/discord_bot.md --- # Create a Discord Bot --- title: Create a Discord Bot latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: discord_bot page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: discord_bot.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/discord_bot.adoc description: Learn how to use Redpanda Connect to create a Discord bot. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-03-31" --- This cookbook outlines how Redpanda Connect can be used to create a Discord bot for important tasks such as providing insults and bad jokes to your chat. If you’re a member of the [Redpanda Connect Discord server](https://discord.gg/6VaWjzP) then you’re likely already familiar with Blob Bot which is the resulting product. ## [](#consuming-messages)Consuming messages Before you start messing with Redpanda Connect you need to register a new bot with the [Discord Developer Portal](https://discord.com/developers/applications). Start by building an Application, then use the build-a-bot page to choose a bot name and avatar. You should end up with a token generated for the bot, and you’ll also need to add it to your server. As soon as your bot is added to your server and you have a token you can immediately begin consuming messages from a channel with the [`discord` input](../../components/inputs/discord/): ```yaml input: discord: channel_id: ${DISCORD_CHANNEL} bot_token: ${DISCORD_BOT_TOKEN} cache: request_tracking cache_resources: - label: request_tracking file: directory: /tmp/discord_bot ``` > If you aren’t sure how to access the ID of a channel try [this tutorial](https://support.discord.com/hc/en-us/articles/206346498-Where-can-I-find-my-User-Server-Message-ID-). The `poll_period` shouldn’t be too short as it’ll exhaust your rate limits. If you plan to use the bot for hitting multiple Discord APIs then give it a fair few seconds between each poll. It’s also necessary to point the input to a [cache resource](../../components/caches/about/), and this will be used to store the ID of the latest message received for paginating the messages endpoint. The `limit` is the maximum number of messages to consume from the channel when we haven’t got a message to track and are consuming a backlog. The first time we run our bot we will pull a maximum of 10 of the latest messages in the channel, the maximum you can set here is 100. If you were to run this config (setting the channel and bot token as env vars in a file called `testing.env`) you’ll see it print messages from the channel to stdout in JSON form: ```bash rpk connect run -e testing.env ./config.yaml {"content":"so i like totally just tripped over my own network cables","author":{"id":"1234"}} {"content":"like omg that is SO you!!!","author":{"id":"4321"}} {"content":"yas totally","author":{"id":"1234"}} {"content":"yeah","author":{"id":"4321"}} ``` ## [](#writing-messages)Writing messages Writing messages to a Discord channel is pretty easy. You can feed the [`discord` output](../../components/outputs/discord/) either a JSON object following the [Message Object structure](https://discord.com/developers/docs/resources/channel#message-object), or just a raw string and the structure will be created for you. Therefore we can write a hypothetical uppercasing echo bot with a simple [Bloblang mapping](../../guides/bloblang/about/): ```yaml pipeline: processors: - mapping: | root = if !this.content.has_prefix("SHOUTS BACK") { "SHOUTS BACK BOT SAYS " + this.content.uppercase() } else { deleted() } output: discord: channel_id: ${DISCORD_CHANNEL} bot_token: ${DISCORD_BOT_TOKEN} ``` If we add that to the end of the first config you should see the bot respond to messages in the channel by posting an uppercase version of it with a prefix. Note that we also delete the message in our mapping if it has the same prefix that we’re adding ourselves, which is a quick and dirty way of ensuring the bot doesn’t echo its own messages. ## [](#custom-commands)Custom commands Shout bot is clearly an absolute riot and a true fan favourite. However, it will get old fast. Let’s make our bot more elegant by introducing some commands by swapping our plain mapping with a [`switch` processor](../../components/processors/switch/): ```yaml pipeline: processors: - switch: - check: this.type == 7 processors: - mapping: 'root = "Welcome to the server <@%v>!".format(this.author.id)' - processors: - mapping: 'root = deleted()' ``` By changing our mapping out to this switch we can add specialised commands for different message types, and if none of the cases match then we don’t respond. Technically, we can do all of this within a single Bloblang mapping by using a match expression, but having a switch processor would also allow us to add cases where we do cool things like hit other APIs, etc. The only case we’ve added here is one that activates when the message type is a specific one sent when a new person joins, and in response we give them a warm welcome. The welcome mentions the new user by injecting the user id into the welcome string with `.format(this.author.id)`, which replaces the `%v` placeholder with the author ID (the user that joined and therefore created the join message). This response is good but not very interactive. Let’s add a few commands that people can play with: ```yaml pipeline: processors: - switch: - check: this.type == 7 processors: - mapping: 'root = "Welcome to the server <@%v>!".format(this.author.id)' - check: this.content == "/joke" processors: - mapping: | let jokes = [ "What do you call a belt made of watches? A waist of time.", "What does a clock do when it’s hungry? It goes back four seconds.", "A company is making glass coffins. Whether they’re successful remains to be seen.", ] root = $jokes.index(timestamp_unix_nano() % $jokes.length()) - check: this.content == "/roast" processors: - mapping: | let roasts = [ "If <@%v>'s brain was dynamite, there wouldn’t be enough to blow their hat off.", "Someday you’ll go far <@%v>, and I really hope you stay there.", "I’d give you a nasty look, but you’ve already got one <@%v>.", ] root = $roasts.index(timestamp_unix_nano() % $roasts.length()).format(this.author.id) - processors: - mapping: 'root = deleted()' ``` Here we have two new commands. If someone posts a message "/joke" then we respond by selecting one of several exceptionally funny jokes from a static list in the mapping. The second new command is "/roast" and is exclusively for brave souls as the responses can be cruel and personal. The command works similarly to "/joke" with the exception being the ID of the user that made the command will be injected into the roast, as mentioning the target of the roast makes it significantly more heartbreaking (as intended). ## [](#hitting-other-apis)Hitting other APIs Clicking websites and browsing the internet is very difficult and most people are simply too busy for it, it’d therefore be useful if we could have our bot do some browsing for us occasionally. The final command we’re going to add to our bot is "/release", where it will hit the Github API and find out for us what the latest Redpanda Connect release is: ```yaml pipeline: processors: - switch: # Other cases omitted for brevity - check: this.content == "/release" processors: - mapping: 'root = ""' - try: - http: url: https://api.github.com/repos/redpanda-data/connect/releases/latest verb: GET - mapping: 'root = "The latest release of Redpanda Connect is %v: %v".format(this.tag_name, this.html_url)' - catch: - log: fields_mapping: 'root.error = error()' message: "Failed to process message" - mapping: 'root = "Sorry, my circuits are all bent from twerking and I must have malfunctioned."' ``` Here we’ve added a switch case that clears the contents of the message, hits the Github API to obtain the latest Redpanda Connect release as a JSON object, and finally maps the tag name and the URL of the release to a useful message. > We’re hitting the Github API with the [generic `http` processor](../../components/processors/http/), which can be configured to work with most HTTP based APIs. In fact, the Discord input and output are actually [configuration templates](../../configuration/templating/) that use the generic HTTP components [under the hood](https://github.com/redpanda-data/connect/blob/master/template/outputs/discord.yaml). Since this command is networked and therefore has a chance of failure we’ve added some [error handling](../../configuration/error_handling/) mechanisms after the switch processor so that it’d capture errors from this new case and any new cases we add later. Within the catch block we simply log the error for the admin to peruse and change the response message out for a generic "whoopsie daisy" apology. The full config for Blob Bot (with some super secret responses redacted) can be found [in the Github repo](https://github.com/redpanda-data/connect/blob/master/config/examples/discord_bot.yaml). To find out more about Bloblang check out [the guide page](../../guides/bloblang/about/). To find out more about config templates check out the [templates documentation page](../../configuration/templating/). If you want to play with Blob Bot then [join our Discord](https://discord.gg/6VaWjzP). There are also some humans in there that will help you manage your disappointment when you see Blob Bot in action. --- # Page 347: DynamoDB CDC Patterns **URL**: https://docs.redpanda.com/redpanda-connect/cookbooks/dynamodb_cdc.md --- # DynamoDB CDC Patterns --- title: DynamoDB CDC Patterns latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: dynamodb_cdc page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: dynamodb_cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/dynamodb_cdc.adoc description: Learn how to capture, filter, transform, and route DynamoDB change data capture (CDC) events with Redpanda Connect. page-topic-type: cookbook personas: streaming_developer, data_engineer learning-objective-1: Find reusable patterns for capturing DynamoDB CDC events learning-objective-2: Look up integration patterns for routing CDC data to Redpanda and S3 learning-objective-3: Identify patterns for filtering and transforming change events page-git-created-date: "2026-03-02" page-git-modified-date: "2026-03-03" --- The DynamoDB CDC input enables capturing item-level changes from DynamoDB tables with streams enabled. This cookbook provides reusable patterns for filtering, transforming, and routing DynamoDB CDC events to Redpanda, S3, and other destinations. Use this cookbook to: - Find reusable patterns for capturing DynamoDB CDC events - Look up integration patterns for routing CDC data to Redpanda and S3 - Identify patterns for filtering and transforming change events ## [](#prerequisites)Prerequisites Before using these patterns, ensure you have the following configured: ### [](#redpanda-cli)Redpanda CLI Install the Redpanda CLI (`rpk`) to run Redpanda Connect. See [Get Started with Redpanda Connect using `rpk`](../../get-started/quickstarts/rpk/) for installation instructions. ### [](#dynamodb-streams)DynamoDB Streams The source DynamoDB table must have [DynamoDB Streams](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html) enabled with an appropriate view type: - `KEYS_ONLY`: Only the key attributes of the modified item - `NEW_IMAGE`: The entire item as it appears after the modification - `OLD_IMAGE`: The entire item as it appeared before the modification - `NEW_AND_OLD_IMAGES`: Both the new and old item images (recommended for detecting changes) To enable streams on an existing table using the AWS CLI: ```bash aws dynamodb update-table \ --table-name my-table \ --stream-specification StreamEnabled=true,StreamViewType=NEW_AND_OLD_IMAGES ``` ### [](#environment-variables)Environment variables The examples in this cookbook use environment variables for AWS configuration. This allows you to keep credentials secure and separate from your pipeline configuration files. ```bash export DYNAMODB_TABLE=my-table (1) export AWS_REGION=us-east-1 (2) export REDPANDA_BROKERS=localhost:9092 (3) export S3_BUCKET=my-cdc-bucket (4) ``` | 1 | The name of the DynamoDB table with streams enabled. | | --- | --- | | 2 | The AWS region where your DynamoDB table is located. | | 3 | The Redpanda broker addresses (for Redpanda output examples). | | 4 | The S3 bucket name (for S3 output examples). | Redpanda Connect loads AWS credentials from the standard [credential chain](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html) (environment variables, `~/.aws/credentials`, or IAM roles). ## [](#capture-cdc-events)Capture CDC events The simplest pattern captures all change events from a DynamoDB table and outputs them with metadata: ```yaml input: aws_dynamodb_cdc: tables: ["${DYNAMODB_TABLE}"] region: ${AWS_REGION} checkpoint_table: redpanda_dynamodb_checkpoints start_from: trim_horizon pipeline: processors: # Extract the change event details - mapping: | root.event_type = this.eventName root.table = this.tableName root.event_id = this.eventID root.keys = this.dynamodb.keys root.new_image = this.dynamodb.newImage root.old_image = this.dynamodb.oldImage root.sequence_number = this.dynamodb.sequenceNumber root.timestamp = now() output: stdout: codec: lines ``` For details on the CDC event message structure and available fields for Bloblang mappings, see the [message structure](../../components/inputs/aws_dynamodb_cdc/#_message_structure) section in the connector reference. ## [](#filter-cdc-events)Filter CDC events You can filter events to process only specific change types: ```yaml input: aws_dynamodb_cdc: tables: ["${DYNAMODB_TABLE}"] region: ${AWS_REGION} start_from: latest pipeline: processors: # Filter to only process INSERT and MODIFY events (ignore REMOVE) - mapping: | root = if this.eventName == "REMOVE" { deleted() } else { this } # Transform to a simplified format - mapping: | root.event_type = this.eventName root.keys = this.dynamodb.keys root.new_data = this.dynamodb.newImage root.old_data = this.dynamodb.oldImage output: stdout: codec: lines ``` This example: - Filters out `REMOVE` events using `deleted()` - Transforms the event to a simplified format ## [](#route-to-redpanda)Route to Redpanda Stream DynamoDB changes to Redpanda for real-time processing: ```yaml input: aws_dynamodb_cdc: tables: ["${DYNAMODB_TABLE}"] region: ${AWS_REGION} checkpoint_table: redpanda_dynamodb_checkpoints batch_size: 100 poll_interval: 500ms pipeline: processors: # Transform to a Kafka-friendly format with a composite key - mapping: | let keys = this.dynamodb.keys meta kafka_key = [$keys.pk, $keys.sk].filter(v -> v != null).join("#") root.event_type = this.eventName root.table = this.tableName root.timestamp = now() root.keys = this.dynamodb.keys root.new_image = this.dynamodb.newImage root.old_image = this.dynamodb.oldImage output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: dynamodb-cdc-events key: ${! @kafka_key } partitioner: murmur2_hash compression: snappy batching: count: 100 period: 1s ``` This example: - Creates a composite message key from the DynamoDB primary key - Transforms the DynamoDB format to plain JSON - Batches messages for efficient delivery ## [](#route-to-s3)Route to S3 Archive CDC events to S3 for long-term storage and analytics: ```yaml input: aws_dynamodb_cdc: tables: ["${DYNAMODB_TABLE}"] region: ${AWS_REGION} checkpoint_table: redpanda_dynamodb_checkpoints start_from: trim_horizon pipeline: processors: # Add partitioning metadata for S3 organization - mapping: | let event_time = now() meta s3_path = "year=%s/month=%s/day=%s/hour=%s".format( $event_time.ts_format("2006"), $event_time.ts_format("01"), $event_time.ts_format("02"), $event_time.ts_format("15") ) root.event_type = this.eventName root.table = this.tableName root.sequence_number = this.dynamodb.sequenceNumber root.event_time = $event_time root.keys = this.dynamodb.keys root.new_image = this.dynamodb.newImage root.old_image = this.dynamodb.oldImage output: aws_s3: bucket: ${S3_BUCKET} path: dynamodb-cdc/${DYNAMODB_TABLE}/${! @s3_path }/${! uuid_v4() }.json region: ${AWS_REGION} batching: count: 1000 period: 1m processors: - archive: format: lines ``` This example: - Organizes files by time-based partitions (year/month/day/hour) - Batches events and archives them as newline-delimited JSON - Uses UUID file names to prevent collisions ## [](#route-by-event-type)Route by event type Route different event types to different destinations: ```yaml input: aws_dynamodb_cdc: tables: ["${DYNAMODB_TABLE}"] region: ${AWS_REGION} pipeline: processors: # Transform to a common format - mapping: | root.event_type = this.eventName root.table = this.tableName root.timestamp = now() root.keys = this.dynamodb.keys root.data = if this.dynamodb.exists("newImage") { this.dynamodb.newImage } else { this.dynamodb.oldImage } output: switch: cases: # Route INSERT events to a topic for new records - check: this.event_type == "INSERT" output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: dynamodb-inserts # Route MODIFY events to a topic for updates - check: this.event_type == "MODIFY" output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: dynamodb-updates # Route REMOVE events to a topic for deletes - check: this.event_type == "REMOVE" output: redpanda: seed_brokers: - ${REDPANDA_BROKERS} topic: dynamodb-deletes # Fallback for any unexpected event types - output: drop: {} ``` This pattern: - Separates processing pipelines for inserts, updates, and deletes - Applies different retention policies per event type - Enables specialized downstream consumers ## [](#detect-changed-fields)Detect changed fields Compare old and new images to identify which fields changed: ```yaml input: aws_dynamodb_cdc: tables: ["${DYNAMODB_TABLE}"] region: ${AWS_REGION} pipeline: processors: # Only process MODIFY events - mapping: | root = if this.eventName != "MODIFY" { deleted() } else { this } # Compare old and new images to find changed fields - mapping: | let old_data = this.dynamodb.oldImage let new_data = this.dynamodb.newImage root.table = this.tableName root.keys = this.dynamodb.keys root.timestamp = now() # Find fields that changed by comparing key-value pairs root.changes = $new_data.key_values().filter(kv -> !$old_data.exists(kv.key) || $old_data.get(kv.key) != kv.value).map_each(kv -> {"field": kv.key, "old_value": if $old_data.exists(kv.key) { $old_data.get(kv.key) } else { null }, "new_value": kv.value}) # Find fields that were removed root.removed_fields = $old_data.keys().filter(k -> !$new_data.exists(k)) output: stdout: codec: lines ``` This pattern: - Filters to only MODIFY events - Compares old and new images to find differences - Outputs a list of changed fields with their old and new values > 📝 **NOTE** > > This pattern requires the `NEW_AND_OLD_IMAGES` stream view type. The `.key_values()` method converts an object to an array of key-value pairs that can be filtered and mapped. ## [](#checkpointing)Checkpointing The DynamoDB CDC input automatically manages checkpoints in a separate DynamoDB table: ```yaml input: aws_dynamodb_cdc: table: my-table checkpoint_table: my-app-checkpoints (1) checkpoint_limit: 500 (2) start_from: trim_horizon (3) ``` | 1 | Custom checkpoint table name (default: redpanda_dynamodb_checkpoints). | | --- | --- | | 2 | Checkpoint after every 500 messages (lower = better recovery, higher = fewer writes). | | 3 | Start from the oldest available record when no checkpoint exists. | If a checkpoint table doesn’t exist, it’s created automatically with the required schema. ## [](#performance-tuning)Performance tuning Optimize throughput and latency with these settings: ```yaml input: aws_dynamodb_cdc: table: my-table batch_size: 1000 (1) poll_interval: 100ms (2) max_tracked_shards: 10000 (3) throttle_backoff: 50ms (4) ``` | 1 | Maximum records per shard per request (1-1000). | | --- | --- | | 2 | Time between polls when no records are available. | | 3 | Maximum shards to track (for very large tables). | | 4 | Backpressure delay when too many messages are in-flight. | ### [](#throughput-considerations)Throughput considerations - DynamoDB Streams allows 5 `GetRecords` calls per second per shard - Higher `batch_size` improves throughput but increases memory usage - Shorter `poll_interval` reduces latency but increases API calls ## [](#troubleshoot)Troubleshoot ### [](#no-events-received)No events received If you’re not receiving events: 1. Verify streams are enabled on the table: ```bash aws dynamodb describe-table --table-name my-table \ --query 'Table.StreamSpecification' ``` 2. Check that changes are being made to the table 3. Verify `start_from` is set to `trim_horizon` to capture existing stream data ### [](#duplicate-events)Duplicate events Each stream record appears exactly once in DynamoDB Streams. However, if your pipeline fails before checkpointing, records may be re-read on restart, resulting in at-least-once processing semantics. To handle potential duplicates: - Use idempotent processing in downstream systems - Deduplicate using the `dynamodb_sequence_number` metadata - Lower `checkpoint_limit` to reduce the window of possible duplicates ### [](#stream-retention)Stream retention DynamoDB Streams retains data for 24 hours. If your pipeline is offline longer than that: - Consider using [Kinesis Data Streams for DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/kds.html) with the [`aws_kinesis` input](../../components/inputs/aws_kinesis/) instead (up to 1 year retention) - Implement a full-table scan fallback for disaster recovery ## [](#suggested-reading)Suggested reading - [DynamoDB CDC Input Reference](../../components/inputs/aws_dynamodb_cdc/) - [AWS Configuration Guide](../../guides/cloud/aws/) - [Kinesis Input](../../components/inputs/aws_kinesis/) (for Kinesis Data Streams for DynamoDB) - [DynamoDB Streams Documentation](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html) ## Suggested labs - [Stream Jira Issues to Redpanda for Real-Time Metrics](/redpanda-labs/docker-compose/jira-metrics-pipeline/) [Search all labs](/redpanda-labs) --- # Page 348: Enrichment Workflows **URL**: https://docs.redpanda.com/redpanda-connect/cookbooks/enrichments.md --- # Enrichment Workflows --- title: Enrichment Workflows latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: enrichments page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: enrichments.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/enrichments.adoc description: How to configure Redpanda Connect to process a workflow of enrichment services. page-git-created-date: "2024-05-24" page-git-modified-date: "2025-04-10" --- This cookbook demonstrates how to enrich a stream of JSON documents with HTTP services. This method also works with [AWS Lambda functions](../../components/processors/aws_lambda/), [subprocesses](../../components/processors/subprocess/), etc. We will start off by configuring a single enrichment, then we will move onto a workflow of enrichments with a network of dependencies using the [`workflow` processor](../../components/processors/workflow/). Each enrichment will be performed in parallel across a [pre-batched](../../configuration/batching/) stream of documents. Workflow enrichments that do not depend on each other will also be performed in parallel, making this orchestration method very efficient. The imaginary problem we are going to solve is applying a set of NLP based enrichments to a feed of articles in order to detect fake news. We will be consuming and writing to Kafka, but the example works with any [input](../../components/inputs/about/) and [output](../../components/outputs/about/) combination. Articles are received over the topic `articles` and look like this: ```json { "type": "article", "article": { "id": "123foo", "title": "Dogs Stop Barking", "content": "The world was shocked this morning to find that all dogs have stopped barking." } } ``` ## [](#meet-the-enrichments)Meet the enrichments ### [](#claims-detector)Claims detector To start us off we will configure a single enrichment, which is an imaginary 'claims detector' service. This is an HTTP service that wraps a trained machine learning model to extract claims that are made within a body of text. The service expects a `POST` request with JSON payload of the form: ```json { "text": "The world was shocked this morning to find that all dogs have stopped barking." } ``` And returns a JSON payload of the form: ```json { "claims": [ { "entity": "world", "claim": "shocked" }, { "entity": "dogs", "claim": "NOT barking" } ] } ``` Since each request only applies to a single document we will make this enrichment scale by deploying multiple HTTP services and hitting those instances in parallel across our document batches. In order to send a mapped request and map the response back into the original document we will use the [`branch` processor](../../components/processors/branch/), with a child [`http`](../../components/processors/http/) processor. ```yaml input: kafka: addresses: [ TODO ] topics: [ articles ] consumer_group: benthos_articles_group batching: count: 20 # Tune this to set the size of our document batches. period: 1s pipeline: processors: - branch: request_map: 'root.text = this.article.content' processors: - http: url: http://localhost:4197/claims verb: POST result_map: 'root.tmp.claims = this.claims' output: kafka: addresses: [ TODO ] topic: comments_hydrated ``` With this pipeline our documents will come out looking something like this: ```json { "type": "article", "article": { "id": "123foo", "title": "Dogs Stop Barking", "content": "The world was shocked this morning to find that all dogs have stopped barking." }, "tmp": { "claims": [ { "entity": "world", "claim": "shocked" }, { "entity": "dogs", "claim": "NOT barking" } ] } } ``` ### [](#hyperbole-detector)Hyperbole detector Next up is a 'hyperbole detector' that takes a `POST` request containing the article contents and returns a hyperbole score between 0 and 1. This time the format is array-based and therefore supports calculating multiple documents in a single request, making better use of the host machines GPU. A request should take the following form: ```json [ { "text": "The world was shocked this morning to find that all dogs have stopped barking." } ] ``` And the response looks like this: ```json [ { "hyperbole_rank": 0.73 } ] ``` In order to create a single request from a batch of documents, and subsequently map the result back into our batch, we will use the [`archive`](../../components/processors/archive/) and [`unarchive`](../../components/processors/unarchive/) processors in our [`branch`](../../components/processors/branch/) flow, like this: ```yaml pipeline: processors: - branch: request_map: 'root.text = this.article.content' processors: - archive: format: json_array - http: url: http://localhost:4198/hyperbole verb: POST - unarchive: format: json_array result_map: 'root.tmp.hyperbole_rank = this.hyperbole_rank' ``` The purpose of the `json_array` format `archive` processor is to take a batch of JSON documents and place them into a single document as an array. Subsequently, we then send one single request for each batch. After the request is made we do the opposite with the `unarchive` processor in order to convert it back into a batch of the original size. ### [](#fake-news-detector)Fake news detector Finally, we are going to use a 'fake news detector' that takes the article contents as well as the output of the previous two enrichments and calculates a fake news rank between 0 and 1. This service behaves similarly to the claims detector service and takes a document of the form: ```json { "text": "The world was shocked this morning to find that all dogs have stopped barking.", "hyperbole_rank": 0.73, "claims": [ { "entity": "world", "claim": "shocked" }, { "entity": "dogs", "claim": "NOT barking" } ] } ``` And returns an object of the form: ```json { "fake_news_rank": 0.893 } ``` We then wish to map the field `fake_news_rank` from that result into the original document at the path `article.fake_news_score`. Our [`branch`](../../components/processors/branch/) block for this enrichment would look like this: ```yaml pipeline: processors: - branch: request_map: | root.text = this.article.content root.claims = this.tmp.claims root.hyperbole_rank = this.tmp.hyperbole_rank processors: - http: url: http://localhost:4199/fakenews verb: POST result_map: 'root.article.fake_news_score = this.fake_news_rank' ``` Note that in our `request_map` we are targeting fields that are populated from the previous two enrichments. If we were to execute all three enrichments in a sequence we’ll end up with a document looking like this: ```json { "type": "article", "article": { "id": "123foo", "title": "Dogs Stop Barking", "content": "The world was shocked this morning to find that all dogs have stopped barking.", "fake_news_score": 0.76 }, "tmp": { "hyperbole_rank": 0.34, "claims": [ { "entity": "world", "claim": "shocked" }, { "entity": "dogs", "claim": "NOT barking" } ] } } ``` Great! However, as a streaming pipeline this set up isn’t ideal as our first two enrichments are independent and could potentially be executed in parallel in order to reduce processing latency. ## [](#combining-into-a-workflow)Combining into a workflow If we configure our enrichments within a [`workflow` processor](../../components/processors/workflow/) we can use Redpanda Connect to automatically detect our dependency graph, giving us two key benefits: 1. Enrichments at the same level of a dependency graph (claims and hyperbole) will be executed in parallel. 2. When introducing more enrichments to our pipeline the added complexity of resolving the dependency graph is handled automatically by Redpanda Connect. Placing our branches within a [`workflow` processor](../../components/processors/workflow/) makes our final pipeline configuration look like this: ```yaml input: kafka: addresses: [ TODO ] topics: [ articles ] consumer_group: benthos_articles_group batching: count: 20 # Tune this to set the size of our document batches. period: 1s pipeline: processors: - workflow: meta_path: '' # Don't bother storing branch metadata. branches: claims: request_map: 'root.text = this.article.content' processors: - http: url: http://localhost:4197/claims verb: POST result_map: 'root.tmp.claims = this.claims' hyperbole: request_map: 'root.text = this.article.content' processors: - archive: format: json_array - http: url: http://localhost:4198/hyperbole verb: POST - unarchive: format: json_array result_map: 'root.tmp.hyperbole_rank = this.hyperbole_rank' fake_news: request_map: | root.text = this.article.content root.claims = this.tmp.claims root.hyperbole_rank = this.tmp.hyperbole_rank processors: - http: url: http://localhost:4199/fakenews verb: POST result_map: 'root.article.fake_news_score = this.fake_news_rank' - catch: - log: fields_mapping: 'root.content = content().string()' message: "Enrichments failed due to: ${!error()}" - mapping: | root = this root.tmp = deleted() output: kafka: addresses: [ TODO ] topic: comments_hydrated ``` Since the contents of `tmp` won’t be required downstream we remove it after our enrichments using a [`mapping` processor](../../components/processors/mapping/). A [`catch`](../../components/processors/catch/) processor was added at the end of the pipeline which catches documents that failed enrichment. You can replace the log event with a wide range of recovery actions such as sending to a dead-letter/retry queue, dropping the message entirely, etc. You can read more about error handling [in this article](../../configuration/error_handling/). --- # Page 349: Filtering and Sampling **URL**: https://docs.redpanda.com/redpanda-connect/cookbooks/filtering.md --- # Filtering and Sampling --- title: Filtering and Sampling latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: filtering page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: filtering.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/filtering.adoc description: Configure Redpanda Connect to conditionally drop messages. page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- Filtering events in Redpanda Connect is both easy and flexible, this cookbook demonstrates a few different types of filtering you can do. All of these examples make use of the [`mapping` processor](../../components/processors/mapping/) but shouldn’t require any prior knowledge. ## [](#the-basic-filter)The basic filter Dropping events with [Bloblang](../../guides/bloblang/about/) is done by mapping the function `deleted()` to the `root` of the mapped document. To remove all events indiscriminately you can simply do: ```yaml pipeline: processors: - mapping: root = deleted() ``` But that’s most likely not what you want. We can instead only delete an event under certain conditions with a [`match`](../../guides/bloblang/about/#pattern-matching) or [`if`](../../guides/bloblang/about/#conditional-mapping) expression: ```yaml pipeline: processors: - mapping: | root = if @topic.or("") == "foo" || this.doc.type == "bar" || this.doc.urls.contains("https://www.benthos.dev/").catch(false) { deleted() } ``` The above config removes any events where: - The metadata field `topic` is equal to `foo` - The event field `doc.type` (a string) is equal to `bar` - The event field `doc.urls` (an array) contains the string `https://www.benthos.dev/` Events that do not match any of these conditions will remain unchanged. ## [](#sample-events)Sample events Another type of filter we might want is a sampling filter, we can do that with a random number generator: ```yaml pipeline: processors: - mapping: | # Drop 50% of documents randomly root = if random_int() % 2 == 0 { deleted() } ``` We can also do this in a deterministic way by hashing events and filtering by that hash value: ```yaml pipeline: processors: - mapping: | # Drop ~10% of documents deterministically (same docs filtered each run) root = if content().hash("xxhash64").slice(-8).number() % 10 == 0 { deleted() } ``` --- # Page 350: Work with Jira Issues **URL**: https://docs.redpanda.com/redpanda-connect/cookbooks/jira.md --- # Work with Jira Issues --- title: Work with Jira Issues latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: jira page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: jira.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/jira.adoc description: Learn how to use the Jira processor as both an input and output in Redpanda Connect. page-git-created-date: "2025-11-21" page-git-modified-date: "2026-01-05" --- The Jira processor enables querying Jira issues using JQL (Jira Query Language) and returning structured data. Unlike traditional connectors, Jira is implemented as a **processor only**, which can be used as both an input (with `generate`) and an output (with `drop`). ## [](#prerequisites)Prerequisites ### [](#redpanda-cli)Redpanda CLI Install the Redpanda CLI (`rpk`) to run Redpanda Connect. See [Get Started with Redpanda Connect using `rpk`](../../get-started/quickstarts/rpk/) for installation instructions. ### [](#enterprise-license)Enterprise license The Jira processor requires a Redpanda Enterprise Edition license. See [Enterprise Licensing](../../get-started/licensing/) for instructions on generating a trial license and setting the `REDPANDA_LICENSE` environment variable. ### [](#jira-environment-variables)Jira environment variables The examples in this cookbook use environment variables for Jira credentials and configuration. This allows you to keep sensitive credentials secure and separate from your pipeline configuration files. 1. [Generate a Jira API token](https://id.atlassian.com/manage-profile/security/api-tokens). 2. Set up your environment variables: ```bash JIRA_BASE_URL=https://.atlassian.net (1) JIRA_USERNAME= (2) JIRA_API_TOKEN= (3) JIRA_PROJECT= (4) ``` | 1 | Your Jira instance URL. You can find this in your browser when logged into Jira. | | --- | --- | | 2 | Your Jira account email address. | | 3 | The API token generated from your Atlassian account. | | 4 | The key of the Jira project you want to query (such as DOC or K8S). | You can export these variables in your shell or place them in a `.env` file for use with `rpk connect run --env-file `. ## [](#use-jira-as-an-input)Use Jira as an input To use Jira as an input, combine the `generate` input with the Jira processor. This pattern triggers Jira queries at regular intervals or on-demand. ### [](#query-jira-periodically)Query Jira periodically This example queries Jira every 30 seconds for recent issues: ```yaml input: generate: mapping: | root.jql = "project = ${JIRA_PROJECT} AND created >= -1d ORDER BY created DESC" root.maxResults = 50 root.fields = ["key", "summary", "status", "assignee", "created"] interval: 30s pipeline: processors: - jira: base_url: "${JIRA_BASE_URL}" username: "${JIRA_USERNAME}" api_token: "${JIRA_API_TOKEN}" max_results_per_page: 50 timeout: 30s backoff: max_retries: 10 # Transform each issue - mapping: | root.issue_key = this.key root.summary = this.fields.summary root.status = this.fields.status.name root.assignee = if this.fields.assignee != null { this.fields.assignee.displayName } else { "Unassigned" } root.created = this.fields.created output: stdout: codec: lines ``` ### [](#one-time-query)One-time query For a single query, use `count` instead of `interval`: ```yaml input: generate: mapping: | root.jql = "project = ${JIRA_PROJECT} ORDER BY created DESC" root.maxResults = 100 root.fields = ["key", "summary", "status"] count: 1 # Run only once pipeline: processors: - jira: base_url: "${JIRA_BASE_URL}" username: "${JIRA_USERNAME}" api_token: "${JIRA_API_TOKEN}" output: stdout: {} ``` ### [](#advanced-query-with-custom-fields)Advanced query with custom fields ```yaml input: generate: mapping: | root.jql = "assignee = currentUser() AND status IN (Open, 'In Progress') ORDER BY updated DESC" root.maxResults = 20 root.fields = [ "key", "summary", "status", "assignee", "priority", "created", "updated", "customfield_10001" # Custom field ID ] root.startAt = 0 interval: 60s pipeline: processors: - jira: base_url: "${JIRA_BASE_URL}" username: "${JIRA_USERNAME}" api_token: "${JIRA_API_TOKEN}" max_results_per_page: 100 # Transform each individual issue message - mapping: | root.key = this.key root.summary = this.fields.summary root.status = this.fields.status.name root.priority = this.fields.priority.name root.custom_field = this.fields.customfield_10001 output: kafka: addresses: ["localhost:9092"] topic: jira-issues ``` ## [](#use-jira-as-an-output)Use Jira as an output To use Jira as an output, place the Jira processor before a `drop` output. This executes queries without forwarding results downstream. ### [](#query-and-drop-results)Query and drop results ```yaml input: stdin: codec: lines pipeline: processors: # Prepare Jira query from incoming message - mapping: | root.jql = this.jql.or("project = ${JIRA_PROJECT} AND status = Open") root.maxResults = this.maxResults.or(100) root.fields = this.fields.or(["key", "summary", "status"]) # Execute Jira query - jira: base_url: "${JIRA_BASE_URL}" username: "${JIRA_USERNAME}" api_token: "${JIRA_API_TOKEN}" backoff: max_retries: 3 timeout: 30s # Log each issue before dropping - log: message: "Processing Jira issue: ${! this.key } - ${! this.fields.summary }" level: INFO output: drop: {} ``` ### [](#trigger-queries-from-kafka)Trigger queries from Kafka ```yaml input: kafka: addresses: ["localhost:9092"] topics: ["jira-query-requests"] consumer_group: jira-query-consumer pipeline: processors: # Parse incoming query request - mapping: | root.jql = this.query root.maxResults = 50 # Execute query - jira: base_url: "${JIRA_BASE_URL}" username: "${JIRA_USERNAME}" api_token: "${JIRA_API_TOKEN}" # Process each individual issue - mapping: | root.query_id = uuid_v4() root.timestamp = now() root.issue_key = this.key root.issue_summary = this.fields.summary output: drop: {} ``` ## [](#input-message-format)Input message format The Jira processor expects input messages containing valid Jira queries in JSON format: ```json { "jql": "project = MYPROJECT AND status = Open", "maxResults": 50, "fields": ["key", "summary", "status", "assignee"] } ``` ### [](#required-fields)Required fields - `jql`: The JQL (Jira Query Language) query string ### [](#optional-fields)Optional fields - `maxResults`: Maximum number of results to return (default: 50) - `fields`: Array of field names to include in the response ## [](#configuration-options)Configuration options For a complete list of configuration options, see [Jira Processor Reference](../../components/processors/jira/). ## [](#jql-query-language)JQL query language The Jira processor supports querying Jira issues using JQL (Jira Query Language). JQL is a flexible, text-based query language similar to SQL that allows you to search for issues matching specific criteria. ### [](#common-jql-patterns)Common JQL patterns Here are common JQL patterns for filtering issues: #### [](#recent-issues-by-project)Recent issues by project ```jql project = AND created >= -7d ORDER BY created DESC ``` #### [](#issues-assigned-to-current-user)Issues assigned to current user ```jql assignee = currentUser() AND status != Done ``` #### [](#issues-by-status)Issues by status ```jql project = AND status IN (Open, 'In Progress', 'To Do') ``` #### [](#issues-by-priority)Issues by priority ```jql project = AND priority = High ORDER BY created DESC ``` #### [](#issues-updated-recently)Issues updated recently ```jql project = AND updated >= -1d ORDER BY updated DESC ``` ## [](#run-the-examples)Run the examples Each example in this cookbook can be saved as a YAML file and run individually. The examples use environment variables for credentials and configuration, so you don’t need to modify the YAML files. 1. Set up your environment: ```bash # Set Jira credentials export JIRA_BASE_URL=https://your-domain.atlassian.net export JIRA_USERNAME=your-email@example.com export JIRA_API_TOKEN=your-api-token export JIRA_PROJECT=YOUR_PROJECT ``` 2. Save any example to a YAML file (for example, `jira-query.yaml`). 3. Run the example: ```bash rpk connect run jira-query.yaml ``` ## [](#output-message-format)Output message format The Jira processor returns **individual issue messages**, rather than a response object with an `issues` array. Each message output by the Jira processor represents a single issue: ```json { "id": "12345", "key": "DOC-123", "fields": { "summary": "Example issue", "status": { "name": "In Progress" }, "assignee": { "displayName": "John Doe" }, "created": "2025-01-15T10:30:00.000-0800", "updated": "2025-01-16T14:20:00.000-0800" } } ``` ## [](#process-output-messages)Process output messages When processing messages from the Jira processor, keep in mind: - **Do not** use `unarchive` to split an `issues` array - **Do not** reference `this.issues` or `this.total` in your mappings - **Do** access fields directly such as `this.key`, `this.fields.summary`. ### [](#handle-null-fields)Handle null fields Some Jira fields can be null and require conditional handling: ```yaml pipeline: processors: - mapping: | # Assignee can be null root.assignee = if this.fields.assignee != null { this.fields.assignee.displayName } else { "Unassigned" } # Reporter can be null root.reporter = if this.fields.reporter != null { this.fields.reporter.displayName } else { "Unknown" } # Resolution date is null for open issues root.resolved_at = this.fields.resolutiondate # Calculate lead time only for resolved issues root.lead_time_days = if this.fields.resolutiondate != null { (this.fields.resolutiondate.ts_parse("2006-01-02T15:04:05.999-0700").ts_unix() - this.fields.created.ts_parse("2006-01-02T15:04:05.999-0700").ts_unix()) / 86400 } else { null } ``` The Jira processor automatically handles pagination internally using the `max_results_per_page` setting. The processor: 1. Makes the initial request with `startAt=0`. 2. Checks if more results are available. 3. Automatically fetches subsequent pages until all results are retrieved. 4. Outputs each issue as an individual message. You don’t need to handle pagination manually. The processor streams all matching issues as separate messages. ## [](#error-handling)Error handling The Jira processor includes built-in retry logic for rate limiting (HTTP 429) and configurable timeouts. See [Jira Processor Reference](../../components/processors/jira/) for detailed configuration options. ## [](#create-and-update-jira-issues)Create and update Jira issues The Jira processor is **read-only** and only supports querying. To create or update Jira issues, you must use the `http` processor with the Jira REST API. ### [](#create-a-jira-issue)Create a Jira issue Use the HTTP processor to POST to the Jira REST API `/issue` endpoint. This example creates a Task issue with a title and description: ```yaml # Create Jira Issue with HTTP Processor # # The Jira processor is read-only and cannot create or update issues. # Use the HTTP processor to create Jira issues using the REST API. input: generate: mapping: | root = { "title": "Example issue from Redpanda Connect", "description": "This issue was created using the HTTP processor", "issue_type": "Task" } count: 1 pipeline: processors: - mapping: | root = { "fields": { "project": { "key": "${JIRA_PROJECT}" }, "summary": this.title, "description": { "type": "doc", "version": 1, "content": [ { "type": "paragraph", "content": [ { "type": "text", "text": this.description } ] } ] }, "issuetype": { "name": this.issue_type } } } - label: create_issue http: url: "${JIRA_BASE_URL}/rest/api/3/issue" verb: POST headers: Content-Type: application/json Accept: application/json basic_auth: enabled: true username: "${JIRA_USERNAME}" password: "${JIRA_API_TOKEN}" - mapping: | root.issue_key = this.key root.issue_id = this.id root.self_link = this.self output: stdout: codec: lines tests: - name: Transform data for creating issue target_processors: '/pipeline/processors/0' environment: JIRA_PROJECT: "TEST" input_batch: - json_content: title: "Test issue" description: "Test description" issue_type: "Bug" output_batches: - - json_equals: fields: project: key: "TEST" summary: "Test issue" description: type: "doc" version: 1 content: - type: "paragraph" content: - type: "text" text: "Test description" issuetype: name: "Bug" - name: Mock create issue API response target_processors: '/pipeline/processors' mocks: create_issue: mapping: | root.id = "10001" root.key = "DOC-789" root.self = "https://your-domain.atlassian.net/rest/api/3/issue/10001" input_batch: - json_content: title: "Test issue" description: "Test description" issue_type: "Task" output_batches: - - json_equals: issue_key: "DOC-789" issue_id: "10001" self_link: "https://your-domain.atlassian.net/rest/api/3/issue/10001" ``` - **Atlassian Document Format (ADF)**: The `description` field uses ADF, Jira’s rich text format. For simple text, wrap it in a paragraph structure as shown. - **Issue type**: Common types are `Task`, `Bug`, `Story`, `Epic`. The type must exist in your Jira project. - **Required fields**: At minimum, you need `project`, `summary`, and `issuetype`. Other fields may be required based on your Jira configuration. - **Response**: The API returns the created issue’s `key`, `id`, and `self` (URL). ## [](#troubleshoot)Troubleshoot ### [](#license-errors)License errors If you receive license-related errors: 1. Verify the license is exported: ```bash echo $REDPANDA_LICENSE ``` 2. Regenerate if needed: ```bash rpk generate license --name "" \ --last-name "" \ --email "" \ --company "" export REDPANDA_LICENSE=$(cat ./redpanda.license) ``` 3. Check license expiration: The trial license expires after 30 days. Check the output from `rpk generate license` for the expiration date. ### [](#authentication-errors)Authentication errors If you receive 401 Unauthorized errors: - Verify your API token is correct - Ensure you’re using your email address as the username - Check that your Jira instance URL is correct (include `https://`) ### [](#rate-limiting)Rate limiting If you hit Jira rate limits (HTTP 429): - Increase `max_retries` in the Jira processor config - Reduce query frequency by increasing `interval` - Use more specific JQL queries to reduce result sizes ### [](#query-performance)Query performance For large result sets: - Use more specific JQL queries - Limit fields with the `fields` parameter - Reduce `max_results_per_page` if memory is constrained ## [](#see-also)See also - [Jira Processor Reference](../../components/processors/jira/) - [Jira REST API Documentation](https://developer.atlassian.com/cloud/jira/platform/rest/v3/intro/) - [JQL Query Guide](https://www.atlassian.com/software/jira/guides/jql) - [Bloblang Interpolation](../../configuration/interpolation/) ## Suggested labs - [Stream Jira Issues to Redpanda for Real-Time Metrics](/redpanda-labs/docker-compose/jira-metrics-pipeline/) [Search all labs](/redpanda-labs) --- # Page 351: Joining Streams **URL**: https://docs.redpanda.com/redpanda-connect/cookbooks/joining_streams.md --- # Joining Streams --- title: Joining Streams latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: joining_streams page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: joining_streams.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/joining_streams.adoc description: How to hydrate documents by joining multiple streams. page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- This cookbook demonstrates how to merge JSON events from parallel streams using content based rules and a [cache](../../components/caches/about/) of your choice. The imaginary problem we are going to solve is hydrating a feed of article comments with information from their parent articles. We will be consuming and writing to Kafka, but the example works with any [input](../../components/inputs/about/) and [output](../../components/outputs/about/) combination. Articles are received over the topic `articles` and look like this: ```json { "type": "article", "article": { "id": "123foo", "title": "Good article", "content": "this is a totally good article" }, "user": { "id": "user1" } } ``` Comments can either be posted on an article or a parent comment, are received over the topic `comments`, and look like this: ```json { "type": "comment", "comment": { "id": "456bar", "parent_id": "123foo", "content": "this article is bad" }, "user": { "id": "user2" } } ``` Our goal is to end up with a single stream of comments, where information about the root article of the comment is attached to the event. The above comment should exit our pipeline looking like this: ```json { "type": "comment", "comment": { "id": "456bar", "parent_id": "123foo", "content": "this article is bad" }, "article": { "title": "Good article", "content": "this is a totally good article" }, "user": { "id": "user2" } } ``` In order to achieve this we will need to cache articles as they pass through our pipelines and then retrieve them for each comment passing through. Since the parent of a comment might be another comment we will also need to cache and retrieve comments in the same way. ## [](#caching-articles)Caching articles Our first pipeline is very simple, we just consume articles, reduce them to only the fields we wish to cache, and then cache them. If we receive the same article multiple times we’re going to assume it’s okay to overwrite the old article in the cache. In this example I’m targeting Redis, but you can choose any of the supported [cache targets](../../components/caches/about/). The TTL of cached articles is set to one week. ```yaml input: kafka: addresses: [ TODO ] topics: [ articles ] consumer_group: benthos_articles_group pipeline: processors: # Reduce document into only fields we wish to cache. - mapping: 'article = article' # Store reduced articles into our cache. - cache: operator: set resource: hydration_cache key: '${!json("article.id")}' value: '${!content()}' # Drop all articles after they are cached. output: drop: {} cache_resources: - label: hydration_cache redis: url: TODO default_ttl: 168h ``` ## [](#hydrating-comments)Hydrating comments Our second pipeline consumes comments, caches them in case a subsequent comment references them, obtains its parent (article or comment), and attaches the root article to the event before sending it to our output topic `comments_hydrated`. In this config we make use of the [`branch`](../../components/processors/branch/) processor as it allows us to reduce documents into smaller maps for caching and gives us greater control over how results are mapped back into the document. ```yaml input: kafka: addresses: [ TODO ] topics: [ comments ] consumer_group: benthos_comments_group pipeline: processors: # Perform both hydration and caching within a for_each block as this ensures # that a given message of a batch is cached before the next message is # hydrated, ensuring that when a message of the batch has a parent within # the same batch hydration can still work. - for_each: # Attempt to obtain parent event from cache (if the ID exists). - branch: request_map: 'root = this.comment.parent_id | deleted()' processors: - cache: operator: get resource: hydration_cache key: '${!content()}' # And if successful copy it into the field `article`. result_map: 'root.article = this.article' # Reduce comment into only fields we wish to cache. - branch: request_map: | root.comment.id = this.comment.id root.article = this.article processors: # Store reduced comment into our cache. - cache: operator: set resource: hydration_cache key: '${!json("comment.id")}' value: '${!content()}' # No `result_map` since we don't need to map into the original message. # Send resulting documents to our hydrated topic. output: kafka: addresses: [ TODO ] topic: comments_hydrated cache_resources: - label: hydration_cache redis: url: TODO default_ttl: 168h ``` This pipeline satisfies our basic needs but errors aren’t handled at all, meaning intermittent cache connectivity problems that span beyond our cache retries will result in failed documents entering our `comments_hydrated` topic. This is also the case if a comment arrives in our pipeline before its parent. There are [many patterns for error handling](../../configuration/error_handling/) to choose from in Redpanda Connect. In this example we’re going to introduce a delayed retry queue as it enables us to reprocess failed documents after a grace period, which is isolated from our main pipeline. ## [](#adding-a-retry-queue)Adding a retry queue Our retry queue is going to be another topic called `comments_retried`. Since most errors are related to time we will delay retry attempts by storing the current timestamp after a failed request as a metadata field. We will use an input [`broker`](../../components/inputs/broker/) so that we can consume both the `comments` and `comments_retry` topics in the same pipeline. Our config (omitting the caching sections for brevity) now looks like this: ```yaml input: broker: inputs: - kafka: addresses: [ TODO ] topics: [ comments ] consumer_group: benthos_comments_group - kafka: addresses: [ TODO ] topics: [ comments_retry ] consumer_group: benthos_comments_group processors: - for_each: # Calculate time until next retry attempt and sleep for that duration. # This sleep blocks the topic 'comments_retry' but NOT 'comments', # because both topics are consumed independently and these processors # only apply to the 'comments_retry' input. - sleep: duration: '${! 3600 - ( timestamp_unix() - meta("last_attempted").number() ) }s' pipeline: processors: - try: - for_each: # Attempt to obtain parent event from cache. - branch: {} # Omitted # Reduce document into only fields we wish to cache. - branch: {} # Omitted # If we've reached this point then both processors succeeded. - mapping: 'meta output_topic = "comments_hydrated"' - catch: # If we reach here then a processing stage failed. - mapping: | meta output_topic = "comments_retry" meta last_attempted = timestamp_unix() # Send resulting documents either to our hydrated topic or the retry topic. output: kafka: addresses: [ TODO ] topic: '${!meta("output_topic")}' cache_resources: - label: hydration_cache redis: url: TODO default_ttl: 168h ``` You can find a full example [in the project repo](https://github.com/redpanda-data/connect/blob/master/config/examples/joining_streams.yaml), and with this config we can deploy as many instances of Redpanda Connect as we need as the partitions will be balanced across the consumers. --- # Page 352: Oracle Database CDC Patterns **URL**: https://docs.redpanda.com/redpanda-connect/cookbooks/oracledb_cdc.md --- # Oracle Database CDC Patterns --- title: Oracle Database CDC Patterns latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: oracledb_cdc page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: oracledb_cdc.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/oracledb_cdc.adoc description: Learn how to capture, filter, transform, and route Oracle Database change data capture (CDC) events with Redpanda Connect. page-topic-type: cookbook personas: streaming_developer, data_engineer learning-objective-1: Find reusable patterns for capturing Oracle Database CDC events learning-objective-2: Look up integration patterns for routing CDC data to Redpanda and S3 learning-objective-3: Identify patterns for filtering and transforming change events page-git-created-date: "2026-04-10" page-git-modified-date: "2026-04-10" --- The Oracle Database CDC input enables capturing row-level changes from Oracle tables using LogMiner. This cookbook provides reusable patterns for filtering, transforming, and routing Oracle CDC events to Redpanda, S3, and other destinations. Use this cookbook to: - Find reusable patterns for capturing Oracle Database CDC events - Look up integration patterns for routing CDC data to Redpanda and S3 - Identify patterns for filtering and transforming change events ## [](#prerequisites)Prerequisites Before using these patterns, ensure you have the following configured: ### [](#redpanda-cli)Redpanda CLI Install the Redpanda CLI (`rpk`) to run Redpanda Connect. See [Get Started with Redpanda Connect using `rpk`](../../get-started/quickstarts/rpk/) for installation instructions. ### [](#oracle-database-setup)Oracle Database setup The Oracle database must be configured for CDC: #### [](#enable-supplemental-logging)Enable supplemental logging Supplemental logging must be enabled at the database and table level: ```sql -- Enable database-level supplemental logging ALTER DATABASE ADD SUPPLEMENTAL LOG DATA; -- Enable table-level supplemental logging (for each table) ALTER TABLE my_schema.my_table ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS; ``` #### [](#enable-archivelog-mode)Enable archivelog mode The database must be in ARCHIVELOG mode for LogMiner to access redo logs: ```sql -- Check if archivelog is enabled ARCHIVE LOG LIST; -- Enable archivelog mode (if needed) SHUTDOWN IMMEDIATE; STARTUP MOUNT; ALTER DATABASE ARCHIVELOG; ALTER DATABASE OPEN; ``` #### [](#grant-permissions)Grant permissions The database user needs specific permissions to read redo logs: ```sql -- Create user for CDC CREATE USER cdc_user IDENTIFIED BY ; -- Grant necessary permissions GRANT CREATE SESSION TO cdc_user; GRANT SELECT ANY TABLE TO cdc_user; GRANT SELECT_CATALOG_ROLE TO cdc_user; GRANT EXECUTE_CATALOG_ROLE TO cdc_user; GRANT SELECT ANY TRANSACTION TO cdc_user; GRANT LOGMINING TO cdc_user; -- Grant access to specific tables GRANT SELECT ON my_schema.my_table TO cdc_user; -- Create schema for checkpoint table CREATE USER rpcn IDENTIFIED BY ; GRANT CREATE TABLE TO rpcn; GRANT CREATE PROCEDURE TO rpcn; GRANT UNLIMITED TABLESPACE TO rpcn; ``` ### [](#oracle-wallet-for-ssl-connections)Oracle Wallet (for SSL connections) For secure connections using Oracle Wallet: 1. Create a wallet directory: ```bash mkdir -p /opt/oracle/wallet ``` 2. Use Oracle Wallet Manager or `orapki` to create and configure the wallet: ```bash orapki wallet create -wallet /opt/oracle/wallet -auto_login orapki wallet add -wallet /opt/oracle/wallet -trusted_cert -cert ca.crt orapki wallet add -wallet /opt/oracle/wallet -user_cert -cert client.crt -pwd ``` 3. The wallet directory should contain either: - `cwallet.sso` (auto-login wallet, no password required) - `ewallet.p12` (requires password) ### [](#environment-variables)Environment variables The examples in this cookbook use environment variables for Oracle configuration. This allows you to keep credentials secure and separate from your pipeline configuration files. ```bash export ORACLE_CONNECTION_STRING=oracle://cdc_user:password@host:1521/service_name (1) export ORACLE_WALLET_PATH=/opt/oracle/wallet (2) export ORACLE_WALLET_PASSWORD=wallet_password (3) export REDPANDA_BROKERS=localhost:9092 (4) export S3_BUCKET=my-cdc-bucket (5) ``` | 1 | The Oracle connection string with credentials. | | --- | --- | | 2 | Path to Oracle Wallet directory (optional, for SSL). | | 3 | Wallet password if using ewallet.p12 (optional). | | 4 | The Redpanda broker addresses (for Redpanda output examples). | | 5 | The S3 bucket name (for S3 output examples). | ## [](#capture-cdc-events)Capture CDC events The simplest pattern captures all change events from Oracle tables and outputs them with metadata: ```yaml input: oracledb_cdc: connection_string: ${ORACLE_CONNECTION_STRING} include: - "MY_SCHEMA\\.ORDERS" - "MY_SCHEMA\\.CUSTOMERS" exclude: - "MY_SCHEMA\\..*_TEMP" pipeline: processors: # Extract the change event details - mapping: | root = this root.metadata.operation = meta("operation") root.metadata.table = meta("table_name") root.metadata.schema = meta("database_schema") root.metadata.scn = meta("scn") root.metadata.timestamp = now() output: stdout: codec: lines ``` For details on the CDC event message structure and available metadata fields, see the [metadata section](../../components/inputs/oracledb_cdc/#_metadata) in the connector reference. ## [](#filter-cdc-events)Filter CDC events You can filter events to process only specific change types: ```yaml input: oracledb_cdc: connection_string: ${ORACLE_CONNECTION_STRING} include: - "MY_SCHEMA\\.ORDERS" pipeline: processors: # Filter out delete events - mapping: | root = if meta("operation") == "delete" { deleted() } else { this } # Add operation metadata to the message - mapping: | root = this.merge({ "operation": meta("operation"), "table": meta("table_name") }) output: stdout: codec: lines ``` This example: - Filters out `delete` events using `deleted()` - Adds operation type and timestamp to the output - Transforms the event to include metadata ## [](#stream-initial-snapshot)Stream initial snapshot Capture both existing data and ongoing changes: ```yaml input: oracledb_cdc: connection_string: ${ORACLE_CONNECTION_STRING} include: - "MY_SCHEMA\\.ORDERS" stream_snapshot: true snapshot_max_batch_size: 5000 max_parallel_snapshot_tables: 2 pipeline: processors: - mapping: | root = this # Tag snapshot vs CDC events root.event_type = if meta("operation") == "read" { "snapshot" } else { "cdc" } output: stdout: codec: lines ``` | 1 | Enable initial snapshot of existing table data. | | --- | --- | | 2 | Number of rows per batch during snapshot (higher = faster, more memory). | | 3 | Number of tables to snapshot in parallel. | > 📝 **NOTE** > > Snapshot events have `operation` metadata set to `"read"`, while CDC events use `"insert"`, `"update"`, or `"delete"`. ## [](#route-to-redpanda)Route to Redpanda Stream Oracle changes to Redpanda for real-time processing: ```yaml input: oracledb_cdc: connection_string: ${ORACLE_CONNECTION_STRING} include: - "MY_SCHEMA\\.ORDERS" - "MY_SCHEMA\\.CUSTOMERS" pipeline: processors: # Create a composite key from the primary key fields - mapping: | meta kafka_key = this.ID.string() # Add CDC metadata as message headers meta operation = meta("operation") meta table = meta("table_name") meta scn = meta("scn") output: redpanda: seed_brokers: ["${REDPANDA_BROKERS}"] topic: 'oracle-cdc-${! meta("table_name").lowercase() }' max_in_flight: 10 compression: snappy batching: count: 100 period: 100ms ``` | 1 | Route to different topics based on table name. | | --- | --- | This example: - Creates a message key from the primary key for consistent partitioning - Routes each table to a separate topic - Batches messages for efficient delivery - Adds CDC metadata as Kafka headers ## [](#route-to-s3)Route to S3 Archive CDC events to S3 for long-term storage and analytics: ```yaml input: oracledb_cdc: connection_string: ${ORACLE_CONNECTION_STRING} include: - "MY_SCHEMA\\.ORDERS" pipeline: processors: - mapping: | root = this root.operation = meta("operation") root.table = meta("table_name") root.scn = meta("scn") root.captured_at = now().ts_format("2006-01-02T15:04:05Z07:00") output: aws_s3: bucket: ${S3_BUCKET} path: 'oracle-cdc/${! meta("table_name") }/${! timestamp_unix().ts_format("2006/01/02/15") }/${! uuid_v4() }.jsonl' batching: count: 1000 period: 5m processors: - archive: format: json_array - compress: algorithm: gzip ``` | 1 | Organize files by table and time-based partitions (year/month/day/hour). | | --- | --- | | 2 | Archive as JSON array for easier downstream parsing. | | 3 | Compress with gzip to reduce storage costs. | This example: - Organizes files by table name and hourly partitions - Batches events and archives them as compressed JSON - Uses UUID file names to prevent collisions ## [](#route-by-event-type)Route by event type Route different event types to different destinations: ```yaml input: oracledb_cdc: connection_string: ${ORACLE_CONNECTION_STRING} include: - "MY_SCHEMA\\.ORDERS" pipeline: processors: # Add routing metadata - mapping: | root = this root.event_type = meta("operation") root.table = meta("table_name") output: switch: cases: # Route INSERT events - check: this.event_type == "insert" output: redpanda: seed_brokers: ["${REDPANDA_BROKERS}"] topic: orders-inserts # Route UPDATE events - check: this.event_type == "update" output: redpanda: seed_brokers: ["${REDPANDA_BROKERS}"] topic: orders-updates # Route DELETE events - check: this.event_type == "delete" output: redpanda: seed_brokers: ["${REDPANDA_BROKERS}"] topic: orders-deletes # Fallback for snapshot reads - output: drop: {} ``` This pattern: - Separates processing pipelines for inserts, updates, and deletes - Routes each operation type to a dedicated topic - Enables specialized downstream consumers per operation type ## [](#detect-changed-columns)Detect changed columns For update operations, identify which columns changed: ```yaml input: oracledb_cdc: connection_string: ${ORACLE_CONNECTION_STRING} include: - "MY_SCHEMA\\.ORDERS" processors: - mapping: | # Only process update operations root = if meta("operation") != "update" { deleted() } # Extract the update event details # Oracle CDC provides before/after values for updates root.id = this.ID root.table = meta("table_name") root.operation = meta("operation") root.scn = meta("scn") # Include the full after-state root.after = this output: stdout: codec: lines ``` > 📝 **NOTE** > > Oracle CDC captures the full row state after the update. For fine-grained change detection, you may need to maintain a previous state externally or use Oracle’s built-in change tracking features. ## [](#checkpointing)Checkpointing The Oracle CDC input automatically manages checkpoints using a table in the Oracle database: ```yaml input: oracledb_cdc: connection_string: ${ORACLE_CONNECTION_STRING} include: - "MY_SCHEMA\\.ORDERS" checkpoint_cache_table_name: RPCN.CDC_CHECKPOINT_CACHE (1) checkpoint_cache_key: orders_cdc_job (2) checkpoint_limit: 500 (3) ``` | 1 | Table name for storing checkpoint state (created automatically if it doesn’t exist). | | --- | --- | | 2 | Unique key to identify this CDC job (useful when running multiple CDC pipelines). | | 3 | Checkpoint after every 500 messages (lower = better recovery, higher = fewer writes). | The checkpoint table stores the System Change Number (SCN) of the last successfully processed event. On restart, CDC resumes from the last checkpoint. ### [](#using-an-external-cache)Using an external cache For better separation or to use a different database for checkpointing: ```yaml cache_resources: - label: postgres_cache postgres: dsn: postgres://user:pass@postgres-host:5432/checkpoints input: oracledb_cdc: connection_string: ${ORACLE_CONNECTION_STRING} include: - "MY_SCHEMA\\.ORDERS" checkpoint_cache: postgres_cache (1) checkpoint_cache_key: orders_cdc_job ``` | 1 | Use an external cache resource instead of the default Oracle table. | | --- | --- | ## [](#performance-tuning)Performance tuning Optimize throughput and latency with these LogMiner settings: ```yaml input: oracledb_cdc: connection_string: ${ORACLE_CONNECTION_STRING} include: - "MY_SCHEMA\\..*" logminer: scn_window_size: 50000 (1) mining_interval: 100ms (2) backoff_interval: 1s (3) max_transaction_events: 10000 (4) lob_enabled: true (5) checkpoint_limit: 2000 (6) batching: count: 100 period: 1s ``` | 1 | Number of SCNs to process in each LogMiner session (higher = more memory, fewer sessions). | | --- | --- | | 2 | How often to query LogMiner for new changes (lower = less latency, more CPU). | | 3 | Backoff when no changes are detected (reduces CPU usage during idle periods). | | 4 | Maximum events to buffer per transaction (0 = unlimited). | | 5 | Enable capture of LOB (CLOB/BLOB) columns. | | 6 | Number of messages to process before checkpointing. | ### [](#throughput-considerations)Throughput considerations - Larger `scn_window_size` reduces LogMiner overhead but increases memory usage - Shorter `mining_interval` reduces latency but increases database load - Higher `checkpoint_limit` improves throughput but increases recovery time after failures - LOB columns significantly impact performance; disable `lob_enabled` if not needed ## [](#secure-connections-with-oracle-wallet)Secure connections with Oracle Wallet Use Oracle Wallet for SSL/TLS connections: ```yaml input: oracledb_cdc: connection_string: oracle://cdc_user:password@host:1522/service wallet_path: ${ORACLE_WALLET_PATH} (1) wallet_password: ${ORACLE_WALLET_PASSWORD} (2) include: - "MY_SCHEMA\\.ORDERS" ``` | 1 | Path to the wallet directory containing cwallet.sso or ewallet.p12. | | --- | --- | | 2 | Wallet password (only required for ewallet.p12 files). | Alternatively, specify wallet configuration in the connection string: ```yaml input: oracledb_cdc: connection_string: oracle://user:password@host:1522/service?WALLET=/opt/oracle/wallet&SSL=true include: - "MY_SCHEMA\\.ORDERS" ``` ## [](#troubleshoot)Troubleshoot ### [](#no-events-received)No events received If you’re not receiving events: 1. Verify supplemental logging is enabled: ```sql -- Check database-level logging SELECT SUPPLEMENTAL_LOG_DATA_MIN FROM V$DATABASE; -- Check table-level logging SELECT LOG_GROUP_NAME, TABLE_NAME, DECODE(ALWAYS, 'ALWAYS', 'Unconditional', NULL, 'Conditional') ALWAYS FROM DBA_LOG_GROUPS WHERE OWNER = 'MY_SCHEMA' AND TABLE_NAME = 'MY_TABLE'; ``` 2. Check that the database is in ARCHIVELOG mode: ```sql SELECT LOG_MODE FROM V$DATABASE; ``` 3. Verify the user has correct permissions: ```sql SELECT * FROM DBA_SYS_PRIVS WHERE GRANTEE = 'CDC_USER'; ``` 4. Make sure changes are being made to the monitored tables ### [](#slow-event-processing)Slow event processing If CDC is lagging behind: - Increase `scn_window_size` to process more changes per LogMiner session - Reduce `checkpoint_limit` to checkpoint more frequently (helps during restarts) - Disable `lob_enabled` if LOB columns aren’t needed - Check Oracle alert logs for performance issues - Monitor the size of redo logs and archive logs ### [](#connection-issues-with-oracle-wallet)Connection issues with Oracle Wallet If SSL connections fail: 1. Verify wallet files exist and have correct permissions: ```bash ls -la /opt/oracle/wallet ``` 2. Check Oracle TNS configuration 3. Verify the wallet password for `ewallet.p12` files 4. Test the connection using Oracle tools (sqlplus, SQL Developer) ### [](#checkpoint-table-errors)Checkpoint table errors If you see errors related to the checkpoint table: 1. Ensure the `rpcn` schema exists: ```sql CREATE USER rpcn IDENTIFIED BY ; GRANT CREATE TABLE TO rpcn; GRANT CREATE PROCEDURE TO rpcn; GRANT UNLIMITED TABLESPACE TO rpcn; ``` 2. Verify the CDC user has access to the checkpoint table: ```sql GRANT SELECT, INSERT, UPDATE ON rpcn.cdc_checkpoint_cache TO cdc_user; ``` ## [](#suggested-reading)Suggested reading - [Oracle CDC Input Reference](../../components/inputs/oracledb_cdc/) - [Redpanda Output](../../components/outputs/redpanda/) - [AWS S3 Output](../../components/outputs/aws_s3/) - [Oracle LogMiner Documentation](https://docs.oracle.com/en/database/oracle/oracle-database/19/sutil/oracle-logminer-utility.html) - [Configuring Supplemental Logging](https://docs.oracle.com/en/database/oracle/oracle-database/19/sutil/configuring-supplemental-logging.html) ## Suggested labs - [Stream Jira Issues to Redpanda for Real-Time Metrics](/redpanda-labs/docker-compose/jira-metrics-pipeline/) [Search all labs](/redpanda-labs) --- # Page 353: Retrieval-Augmented Generation (RAG) **URL**: https://docs.redpanda.com/redpanda-connect/cookbooks/rag.md --- # Retrieval-Augmented Generation (RAG) --- title: Retrieval-Augmented Generation (RAG) latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: rag page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: rag.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/rag.adoc description: How to configure Redpanda Connect to create a RAG pipeline, using PostgreSQL and PGVector. page-git-created-date: "2024-09-12" page-git-modified-date: "2024-09-12" --- This cookbook shows you how to create a Retrieval-Augmented Generation (RAG) pipeline, using PostgreSQL and [PGVector](https://github.com/pgvector/pgvector). Follow the cookbook to: - Create two Redpanda Connect data pipelines: one for creating vector embeddings with [Ollama](https://ollama.ai) and another for searching the data, using [PGVector](https://github.com/pgvector/pgvector). - Run the two pipelines in parallel using [streams mode](../../guides/streams_mode/about/). ## [](#compute-the-embeddings)Compute the embeddings Start by defining the indexing pipeline, which takes textual data from Redpanda, computes vector embeddings for it, and then write it into a PostgreSQL table with a PGVector index on the embeddings column. ```bash rpk topic create articles echo '{ "type": "article", "article": { "id": "123foo", "title": "Dogs Stop Barking", "content": "The world was shocked this morning to find that all dogs have stopped barking." } }' | rpk topic produce articles -f '%v' ``` Your indexing pipeline can read from the Redpanda topic, using the [`kafka`](../../components/inputs/kafka/) input: ```yaml input: kafka: addresses: [ "${REDPANDA_CLUSTER}" ] topics: [ articles ] consumer_group: rp_connect_articles_group tls: enabled: true sasl: mechanism: SCRAM-SHA-256 user: "${REDPANDA_USER}" password: "${REDPANDA_PASSWORD}" ``` Use [Nomic Embed](https://ollama.com/library/nomic-embed-text) to compute embeddings. Since each request only applies to a single document, you can scale this by making requests in parallel across document batches. To send a mapped request and map the response back into the original document, use the [`branch` processor](../../components/processors/branch/) with a child [`ollama_embeddings`](../../components/processors/ollama_embeddings/) processor. ```yaml pipeline: threads: -1 processors: - branch: request_map: 'root = "search_document: %s\n%s".format(this.article.title, this.article.content)' processors: - ollama_embeddings: model: nomic-embed-text result_map: 'root.article.embeddings = this' ``` With this pipeline, your processed documents should look something like this: ```yaml { "type": "article", "article": { "id": "123foo", "title": "Dogs Stop Barking", "content": "The world was shocked this morning to find that all dogs have stopped barking.", "embeddings": [0.754, 0.19283, 0.231, 0.834], # This vector will actually have 768 dimensions } } ``` Now, try sending this transformed data to PostgreSQL using the [`sql_insert`](../../components/outputs/sql_insert/) output. You can take advantage of the `init_statement` functionality to set up `pgvector` and a table to write the data to. ```yaml output: sql_insert: driver: postgres dsn: "${PG_CONNECTION_STRING}" init_statement: | CREATE EXTENSION IF NOT EXISTS vector; CREATE TABLE IF NOT EXISTS searchable_text ( id varchar(128) PRIMARY KEY, title text NOT NULL, body text NOT NULL, embeddings vector(768) NOT NULL ); CREATE INDEX IF NOT EXISTS text_hnsw_index ON searchable_text USING hnsw (embeddings vector_l2_ops); table: searchable_text columns: ["id", "title", "body", "embeddings"] args_mapping: "[this.article.id, this.article.title, this.article.content, this.article.embeddings.vector()]" ``` Save this pipeline as `indexing.yaml` and run it with `rpk connect run ./indexing.yaml` to make sure your PostgreSQL table is populated with embeddings. ## [](#generating-search-responses)Generating search responses To generate responses to questions with the dataset you’re creating embeddings for, use a [HTTP server input](../../components/inputs/http_server/) to receive the questions. Since you’re running this pipeline in streams mode, the HTTP server uses [endpoints prefixed with stream identifier](../../guides/streams_mode/about/#http-endpoints), which in this case is the same as the filename: `search`. ```yaml input: http_server: path: / allowed_verbs: [ GET ] sync_response: headers: Content-Type: application/json ``` The user query uses the query parameter `q`: `[http://localhost:4195/search?q=question_here](http://localhost:4195/search?q=question_here)`. Since query parameters are exposed as metadata in the `http_server` input, you can reference that in bloblang with `@q`. ```yaml pipeline: processors: - label: compute_embeddings ollama_embeddings: model: nomic-embed-text text: "search_query: %s".format(@q) ``` The payload is the embeddings vector. To fetch the top three most similar documents to your embeddings vector, you can write a PostgreSQL query in the [`sql_raw`](../../components/processors/sql_raw/) processor. ```yaml - sql_raw: driver: "postgres" dsn: "${PG_CONNECTION_STRING}" query: SELECT title, body FROM searchable_text ORDER BY embeddings <-> $1 LIMIT 3 args_mapping: "[ this.vector() ]" ``` With your looked up information, as well as your initial query, you can use the [`ollama_chat` processor](../../components/processors/ollama_chat/) to respond to the user’s question as text. ```yaml - label: generate_response ollama_chat: model: llama3.1 prompt: | Your task is to respond to user queries using the provided information. The user asked: ${! @q } Context: ${!this.map_each(row -> "%s\n%s".format(row.title, row.body)).join("\n\n")} Response: ``` Now that you’ve generated a response, you can send that back to the HTTP server as a response using [`sync_response`](../../components/processors/sync_response/). Then, delete the message using a [bloblang mapping](../../components/processors/mapping/) so that nothing goes to the output. ```yaml - mapping: 'root.response = content().string()' - sync_response: {} - mapping: 'root = deleted()' ``` Both pipelines are ready. Try running both of them using streams mode: `rpk connect streams indexing.yaml search.yaml`. When some documents have been indexed, you can query the system using: ```bash curl -G 'localhost:4195/search' --data-urlencode 'q=what is happening to the dogs?' | jq ``` The output should look something like: ```json { "response": "Everyone in the world woke up today shocked as their beloved pooches were silent - unable to bark." } ``` --- # Page 354: Redpanda Migrator **URL**: https://docs.redpanda.com/redpanda-connect/cookbooks/redpanda_migrator.md --- # Redpanda Migrator --- title: Redpanda Migrator latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: redpanda_migrator page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: redpanda_migrator.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/redpanda_migrator.adoc description: Move your workloads from any Kafka system to Redpanda using a single command. Redpanda Migrator lets you migrate Kafka messages, schemas, and ACLs quickly and efficiently. page-git-created-date: "2024-10-02" page-git-modified-date: "2025-10-24" --- With Redpanda Migrator, you can move your workloads from any Apache Kafka system to Redpanda using a single command. It lets you migrate Kafka messages, schemas, and ACLs quickly and efficiently. Redpanda Connect’s Redpanda Migrator uses the unified migrator components (available in Redpanda Connect 4.67.5+): - [`redpanda_migrator` input](../../components/inputs/redpanda_migrator/) connects to the source Kafka cluster and Schema Registry. - [`redpanda_migrator` output](../../components/outputs/redpanda_migrator/) handles all migration logic including topic creation, schema synchronization, and consumer group offset translation. > 📝 **NOTE** > > If you’re currently using the legacy `redpanda_migrator_bundle` components, see [Migrate to the Unified Redpanda Migrator](../../guides/migration/migrate-unified-redpanda-migrator/) for migration instructions. ## [](#create-the-docker-containers)Create the Docker containers First, you need two clusters. To keep it simple, you can run the [Bitnami Kafka](https://hub.docker.com/r/bitnami/kafka) and [Schema Registry](https://hub.docker.com/r/bitnami/schema-registry) Docker containers for the source cluster and a [Redpanda Docker container](https://hub.docker.com/r/redpandadata/redpanda) for the destination cluster via [Docker Compose](https://docs.docker.com/compose). `docker-compose.yml` ```yaml services: source: image: bitnami/kafka environment: KAFKA_CFG_NODE_ID: 0 KAFKA_CFG_PROCESS_ROLES: controller,broker KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 0@localhost:9093 KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_DOCKER:PLAINTEXT KAFKA_CFG_LISTENERS: PLAINTEXT://0.0.0.0:9092,PLAINTEXT_DOCKER://0.0.0.0:19092,CONTROLLER://:9093 KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_DOCKER://source:19092 KAFKA_CFG_AUTHORIZER_CLASS_NAME: "org.apache.kafka.metadata.authorizer.StandardAuthorizer" KAFKA_CFG_SUPER_USERS: User:redpanda;User:ANONYMOUS ports: - 9092:9092 - 19092:19092 healthcheck: test: [ "CMD", "kafka-topics.sh", "--bootstrap-server=localhost:9092", "--list" ] start_period: 5s interval: 3s init_source: image: bitnami/kafka working_dir: /opt/bitnami/kafka/bin entrypoint: /bin/bash depends_on: source: condition: service_healthy command: > -c "kafka-topics.sh --bootstrap-server source:19092 --create --if-not-exists --topic foo --replication-factor=1 --partitions=2 && kafka-topics.sh --bootstrap-server source:19092 --create --if-not-exists --topic bar --replication-factor=1 --partitions=2 && echo 'Created topics:' && kafka-topics.sh --bootstrap-server source:19092 --list --exclude-internal && kafka-acls.sh --bootstrap-server source:19092 --add --allow-principal User:redpanda --operation Read --topic foo && kafka-acls.sh --bootstrap-server source:19092 --add --deny-principal User:redpanda --operation Read --topic bar echo 'Created ACLs:' && kafka-acls.sh --bootstrap-server source:19092 --list" source_schema_registry: image: bitnami/schema-registry environment: SCHEMA_REGISTRY_KAFKA_BROKERS: PLAINTEXT://source:19092 ports: - 8081:8081 depends_on: source: condition: service_healthy destination: image: redpandadata/redpanda command: - redpanda - start - --node-id 0 - --mode dev-container - --set rpk.additional_start_flags=[--reactor-backend=epoll] - --set redpanda.auto_create_topics_enabled=false - --kafka-addr 0.0.0.0:9093 - --advertise-kafka-addr localhost:9093 - --schema-registry-addr 0.0.0.0:8081 ports: - 8082:8081 - 9093:9093 - 9645:9644 ``` ```bash docker compose -f docker-compose.yml up --force-recreate -V ``` > 📝 **NOTE** > > This command uses an `init` container to create two topics, `foo` and `bar`, each with two partitions and an associated ACL. ## [](#create-schemas)Create schemas When the demo clusters are up and running, use [curl](https://curl.se) to create a schema for each topic in the `source` cluster. ```bash curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"schema": "{\"name\": \"Foo\", \"type\": \"record\", \"fields\": [{\"name\": \"data\", \"type\": \"int\"}]}"}' http://localhost:8081/subjects/foo/versions curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"schema": "{\"name\": \"Bar\", \"type\": \"record\", \"fields\": [{\"name\": \"data\", \"type\": \"int\"}]}"}' http://localhost:8081/subjects/bar/versions ``` ## [](#generate-messages)Generate messages Let’s simulate an application with a producer and consumer actively publishing and reading messages on the `source` cluster. You can use Redpanda Connect to generate some Avro-encoded messages and push them to the two topics from the `source` cluster. `generate_data.yaml` ```yaml http: enabled: false input: sequence: inputs: - generate: mapping: | let msg = counter() root.data = $msg meta kafka_topic = match $msg % 2 { 0 => "foo" 1 => "bar" } interval: 1s count: 0 batch_size: 1 processors: - schema_registry_encode: url: "http://localhost:8081" subject: ${! metadata("kafka_topic") } avro_raw_json: true output: kafka_franz: seed_brokers: [ "localhost:9092" ] topic: ${! @kafka_topic } partitioner: manual partition: ${! random_int(min:0, max:1) } ``` Now, run this command to start the pipeline, and leave it running: ```bash rpk connect run generate_data.yaml ``` Next, add a Redpanda Connect consumer, which reads messages from the `source` cluster topics, and leave it running. This consumer uses the `foobar` consumer group, which is reused in a later step when consuming from the `destination` cluster. `read_data_source.yaml` ```yaml http: enabled: false input: kafka_franz: seed_brokers: [ "localhost:9092" ] topics: - '^[^_]' # Skip topics which start with `_` regexp_topics: true consumer_group: foobar processors: - schema_registry_decode: url: "http://localhost:8081" avro_raw_json: true output: stdout: {} processors: - mapping: | root = this.merge({"count": counter(), "topic": @kafka_topic, "partition": @kafka_partition}) ``` Launch the `source` consumer pipeline, and leave it running: ```bash rpk connect run read_data_source.yaml ``` At this point, the `source` cluster has some data in both `foo` and `bar` topics, and the consumer prints the messages it reads from these topics to `stdout`. ## [](#configure-and-start-redpanda-migrator)Configure and start Redpanda Migrator The unified Redpanda Migrator does the following: - The `redpanda_migrator` input connects to the source Kafka cluster and Schema Registry to consume messages and schema information. - The `redpanda_migrator` output handles all migration logic: - Schema migration: reads schemas from the source Schema Registry and synchronizes them to the destination. - Topic creation: automatically creates destination topics that don’t exist with proper configurations. - ACL migration: migrates access control lists according to the migration rules. - Message streaming: processes and routes messages from source to destination topics. - Consumer group offset translation: maps source consumer group offsets to equivalent destination positions. - If new topics are created in the source cluster while the migrator is running, they are migrated when messages are written to them. ACL migration for topics adheres to the following principles: - `ALLOW WRITE` ACLs for topics are not migrated - `ALLOW ALL` ACLs for topics are downgraded to `ALLOW READ` - Group ACLs are not migrated > 📝 **NOTE** > > Changing topic configurations, such as partition count, isn’t currently supported. Now, use the following unified Redpanda Migrator configuration. See the [`redpanda_migrator` input](../../components/inputs/redpanda_migrator/) and [`redpanda_migrator` output](../../components/outputs/redpanda_migrator/) docs for details. `redpanda_migrator.yaml` ```yaml input: label: "migration_pipeline" (1) redpanda_migrator: # Source Kafka settings seed_brokers: [ "localhost:9092" ] topics: - '^[^_]' # Skip internal topics which start with `_` regexp_topics: true consumer_group: migrator # Source Schema Registry settings schema_registry: url: http://localhost:8081 output: label: "migration_pipeline" (2) redpanda_migrator: # Destination Redpanda settings seed_brokers: [ "localhost:9093" ] # Destination Schema Registry and migration settings schema_registry: url: http://localhost:8082 include_deleted: true translate_ids: true # Consumer group migration settings consumer_groups: enabled: true interval: 30s metrics: prometheus: {} mapping: | meta label = if this == "input_redpanda_migrator_lag" { "source" } ``` > 💡 **TIP** > > Label names must be between 3 and 128 characters and can only contain alphanumeric characters, hyphens, and underscores (`A-Za-z0-9-_`). Launch the unified Redpanda Migrator pipeline, and leave it running: ```bash rpk connect run redpanda_migrator.yaml ``` ## [](#check-the-status-of-migrated-topics)Check the status of migrated topics You can use the Redpanda [`rpk` CLI tool](../../../current/get-started/rpk/) to check which topics and ACLs have been migrated to the `destination` cluster. You can quickly [install `rpk`](../../../current/get-started/rpk-install/) if you don’t already have it. > 📝 **NOTE** > > For now, users require manual migration. However, this step is not required for the current demo. Similarly, roles are specific to Redpanda and, for now, also require manual migration if the `source` cluster is based on Redpanda. ```bash rpk -X brokers=localhost:9093 topic list NAME PARTITIONS REPLICAS _schemas 1 1 bar 2 1 foo 2 1 rpk -X brokers=localhost:9093 security acl list PRINCIPAL HOST RESOURCE-TYPE RESOURCE-NAME RESOURCE-PATTERN-TYPE OPERATION PERMISSION ERROR User:redpanda * TOPIC bar LITERAL READ DENY User:redpanda * TOPIC foo LITERAL READ ALLOW ``` ## [](#check-metrics-to-monitor-progress)Check metrics to monitor progress Redpanda Connect provides a comprehensive suite of metrics in various formats, such as Prometheus, which you can use to monitor its performance in your observability stack. Besides the [standard Redpanda Connect metrics](../../components/metrics/about/#metric-names), the `redpanda_migrator` input also emits an `input_redpanda_migrator_lag` metric for monitoring the migration progress of each topic and partition. ```bash curl http://localhost:4195/metrics ... # HELP input_redpanda_migrator_lag Benthos Gauge metric # TYPE input_redpanda_migrator_lag gauge input_redpanda_migrator_lag{label="source",partition="0",path="root.input.sequence.broker.inputs.0",topic="__consumer_offsets"} 0 input_redpanda_migrator_lag{label="source",partition="0",path="root.input.sequence.broker.inputs.0",topic="bar"} 0 input_redpanda_migrator_lag{label="source",partition="0",path="root.input.sequence.broker.inputs.0",topic="foo"} 0 input_redpanda_migrator_lag{label="source",partition="1",path="root.input.sequence.broker.inputs.0",topic="__consumer_offsets"} 0 input_redpanda_migrator_lag{label="source",partition="1",path="root.input.sequence.broker.inputs.0",topic="bar"} 1 input_redpanda_migrator_lag{label="source",partition="1",path="root.input.sequence.broker.inputs.0",topic="foo"} 0 ... ``` ## [](#read-from-the-migrated-topics)Read from the migrated topics Stop the `read_data_source.yaml` consumer you started earlier and, afterwards, start a similar consumer for the `destination` cluster. Before starting the consumer up on the `destination` cluster, make sure you give the migrator bundle some time to replicate the translated offset. `read_data_destination.yaml` ```yaml http: enabled: false input: kafka_franz: seed_brokers: [ "localhost:9093" ] topics: - '^[^_]' # Skip topics which start with `_` regexp_topics: true consumer_group: foobar processors: - schema_registry_decode: url: "http://localhost:8082" avro_raw_json: true output: stdout: {} processors: - mapping: | root = this.merge({"count": counter(), "topic": @kafka_topic, "partition": @kafka_partition}) ``` Now launch the `destination` consumer pipeline, and leave it running: ```bash rpk connect run read_data_destination.yaml ``` The `source` cluster consumer uses the same `foobar` consumer group. This consumer resumes reading messages from where the `source` consumer left off. Redpanda Migrator performs offset remapping when migrating consumer group offsets to the `destination` cluster. While more sophisticated approaches are possible, Redpanda chose to use a simple timestamp-based approach. So, for each migrated offset, the `destination` cluster is queried to find the latest offset before the received offset timestamp. Redpanda Migrator then writes this offset as the `destination` consumer group offset for the corresponding topic and partition pair. Although the timestamp-based approach doesn’t guarantee exactly-once delivery, it minimizes the likelihood of message duplication and avoids the need for complex and error-prone offset remapping logic. --- # Page 355: Ingest data into Snowflake **URL**: https://docs.redpanda.com/redpanda-connect/cookbooks/snowflake_ingestion.md --- # Ingest data into Snowflake --- title: Ingest data into Snowflake latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: snowflake_ingestion page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: snowflake_ingestion.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/cookbooks/pages/snowflake_ingestion.adoc description: Configure Redpanda Connect to ingest data from a Redpanda topic into Snowflake using Snowpipe Streaming. page-git-created-date: "2025-01-28" page-git-modified-date: "2025-02-24" --- Configure a Redpanda Connect pipeline to generate and write data into a local Redpanda topic, and then ingest that data into [Snowflake](https://www.snowflake.com/en/) using [Snowpipe Streaming](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-overview). See also: [Ingest data into Snowflake](../../../redpanda-cloud/develop/connect/cookbooks/snowflake_ingestion/) using Redpanda Cloud ## [](#prerequisites)Prerequisites - [`rpk` installed](https://docs.redpanda.com/current/get-started/rpk-install/) - A [Snowflake account](https://trial.snowflake.com/) - `openssl` command-line tool ## [](#set-up-your-redpanda-cluster)Set up your Redpanda cluster Run [rpk container start](https://docs.redpanda.com/current/reference/rpk/rpk-container/rpk-container-start/) to create a local Redpanda cluster for development: ```none rpk container start ``` Next, create a `demo_topic` to use as the data source for ingesting data into Snowflake: ```bash rpk topic create demo_topic ``` Create a user with minimal [ACLs](https://docs.redpanda.com/current/manage/security/authorization/acl/) to run the ingestion pipeline into Snowflake: ```bash rpk security user create ingestion_user --password Testing1234 ``` Now that the user exists, give them read permissions to `demo_topic`, as well as full control over any consumer group with the prefix `redpanda_connect`: ```bash rpk security acl create --allow-principal ingestion_user --operation read --topic demo_topic rpk security acl create --allow-principal ingestion_user --resource-pattern-type prefixed --operation all --group redpanda_connect ``` ## [](#set-up-your-snowflake-account)Set up your Snowflake account Log in to your Snowflake account with a user who has the ACCOUNTADMIN role. Then, run the following SQL commands in a worksheet. They set up another user with minimal permissions to write data into a specified database and schema, ready for streaming data to Snowflake. ```sql -- Set default values for multiple variables SET PWD = 'Test1234567'; SET USER = 'STREAMING_USER'; SET DB = 'STREAMING_DB'; SET ROLE = 'REDPANDA_CONNECT'; SET WH = 'STREAMING_WH'; USE ROLE ACCOUNTADMIN; -- Create users CREATE USER IF NOT EXISTS IDENTIFIER($USER) PASSWORD=$PWD COMMENT='STREAMING USER FOR REDPANDA CONNECT'; -- Create roles CREATE OR REPLACE ROLE IDENTIFIER($ROLE); -- Create the destination database and virtual warehouse CREATE DATABASE IF NOT EXISTS IDENTIFIER($DB); USE IDENTIFIER($DB); CREATE OR REPLACE WAREHOUSE IDENTIFIER($WH) WITH WAREHOUSE_SIZE = 'SMALL'; -- Grant privileges GRANT CREATE WAREHOUSE ON ACCOUNT TO ROLE IDENTIFIER($ROLE); GRANT ROLE IDENTIFIER($ROLE) TO USER IDENTIFIER($USER); GRANT OWNERSHIP ON DATABASE IDENTIFIER($DB) TO ROLE IDENTIFIER($ROLE); GRANT USAGE ON WAREHOUSE IDENTIFIER($WH) TO ROLE IDENTIFIER($ROLE); -- Set defaults ALTER USER IDENTIFIER($USER) SET DEFAULT_ROLE=$ROLE; ALTER USER IDENTIFIER($USER) SET DEFAULT_WAREHOUSE=$WH; -- Run the following commands to find your account identifier. Copy it down for later use. -- It will be something like `organization_name-account_name` -- e.g. ykmxgak-wyb52636 WITH HOSTLIST AS (SELECT * FROM TABLE(FLATTEN(INPUT => PARSE_JSON(SYSTEM$allowlist())))) SELECT REPLACE(VALUE:host,'.snowflakecomputing.com','') AS ACCOUNT_IDENTIFIER FROM HOSTLIST WHERE VALUE:type = 'SNOWFLAKE_DEPLOYMENT_REGIONLESS'; ``` ### [](#create-an-rsa-key-pair)Create an RSA key pair Create an [RSA key pair](https://docs.snowflake.com/en/user-guide/key-pair-auth) using `openssl` to authenticate Redpanda Connect to Snowflake. When you’re prompted to give an encryption password, record it for later. ```bash openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -passout pass:Testing123 -out rsa_key.p8 ``` Create a public key. You’re prompted to enter your encryption password. ```bash openssl rsa -in rsa_key.p8 -pubout -passout pass:Testing123 -out rsa_key.pub ``` To register the public key in Snowflake, remove the public key delimiters and output only the base64-encoded portion of the PEM file. Run the following bash command to print it: ```bash cat rsa_key.pub | sed -e '1d' -e '$d' | tr -d '\n' ``` In the Snowflake worksheet, add the output of the bash command you just ran to the following SQL command and execute it: ```sql use role accountadmin; alter user streaming_user set rsa_public_key='< PubKeyWithoutDelimiters >'; ``` ### [](#create-a-schema-using-streaming_user)Create a schema using `streaming_user` Log out of Snowflake and sign back in as the default user (`streaming_user`) with the associated password (default: `Test1234567`). You created these credentials in [Set up your Snowflake account](#set-up-your-snowflake-account). Run the following SQL commands in a worksheet to create a schema (e.g. `STREAMING_SCHEMA`) in the default database (e.g. `STREAMING_DB`): ```sql SET DB = 'STREAMING_DB'; SET SCHEMA = 'STREAMING_SCHEMA'; USE IDENTIFIER($DB); CREATE OR REPLACE SCHEMA IDENTIFIER($SCHEMA); ``` ## [](#create-a-pipeline-from-your-redpanda-cluster-to-snowflake)Create a pipeline from your Redpanda cluster to Snowflake You can now create the pipeline locally and inject [secrets](../../configuration/secrets/) using environment variables. First, create a `connect.yaml` file: ```yaml input: # Read data from the local `demo_topic` kafka_franz: seed_brokers: ["localhost:9092"] topics: ["demo_topic"] consumer_group: "redpanda_connect_to_snowflake" tls: {enabled: true} checkpoint_limit: 4096 sasl: - mechanism: SCRAM-SHA-256 username: ingestion_user password: ${REDPANDA_PASS} # Define a batching policy. For this cookbook, create small batches, # but in a production environment use the largest file size you can. batching: count: 100 # Collect 10 messages before flushing period: 10s # or after 10 seconds, which ever comes first output: snowflake_streaming: # Replace this placeholder with your account identifier account: "< OrgName-AccountName >" user: STREAMING_USER role: REDPANDA_CONNECT database: STREAMING_DB schema: STREAMING_SCHEMA table: STREAMING_DATA # Inject your private key and password private_key_file: rsa_key.p8 private_key_pass: "${SNOWFLAKE_KEY_PASS}" schema_evolution: enabled: true max_in_flight: 1 ``` Now run the pipeline, and any JSON data produced into the topic is streamed into Snowflake with minimal latency. ```bash REDPANDA_PASS=Testing1234 SNOWFLAKE_KEY_PASS=Testing123 rpk connect run ./connect.yaml ``` You now can produce some data using `rpk` to test that everything works: ```bash echo '{"animal":"redpanda","attributes":"cute","age":6}' | rpk topic produce demo_topic -f '%v\n' echo '{"animal":"polar bear","attributes":"cool","age":13}' | rpk topic produce demo_topic -f '%v\n' echo '{"animal":"unicorn","attributes":"rare","age":999}' | rpk topic produce demo_topic -f '%v\n' ``` The data produced into the `demo_topic` is consumed and streamed into Snowflake in seconds. Go back to the Snowflake worksheet and run the following query to see data arrive in Snowflake with the schema from the JSON data you produced. ```sql SELECT * FROM STREAMING_DB.STREAMING_SCHEMA.STREAMING_DATA LIMIT 50; ``` See also: - The [`kafka_franz` input](../../components/inputs/kafka_franz/) - The [`snowflake_streaming`](../../components/outputs/snowflake_streaming/) output --- # Page 356: Get Started **URL**: https://docs.redpanda.com/redpanda-connect/get-started.md --- # Get Started --- title: Get Started latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: index page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: index.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/get-started/pages/index.adoc description: Explore these topics to get started with Redpanda Connect. page-git-created-date: "2024-11-14" page-git-modified-date: "2024-11-14" --- Explore these topics to get started with Redpanda Connect. - [Introduction to Redpanda Connect](about/) Learn about Redpanda Connect, a declarative connector framework for building data pipelines with simple, chained, stateless processing steps. - [What’s New in Redpanda Connect](whats-new/) Summary of new features in Redpanda Connect. - [Quickstarts](quickstarts/) Use your preferred tool to spin up a local instance of Redpanda Connect and get hands-on experience building your first pipelines. - [Enterprise Licensing](licensing/) Learn how to purchase and apply an Enterprise Edition license. - [Redpanda Connect Versioning](versioning/) Learn how Redpanda Connect implements versioning. --- # Page 357: Introduction to Redpanda Connect **URL**: https://docs.redpanda.com/redpanda-connect/get-started/about.md --- # Introduction to Redpanda Connect --- title: Introduction to Redpanda Connect latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: about page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/get-started/pages/about.adoc description: Learn about Redpanda Connect, a declarative connector framework for building data pipelines with simple, chained, stateless processing steps. page-git-created-date: "2024-11-14" page-git-modified-date: "2026-03-31" --- Redpanda Connect is a declarative connector framework that solves a wide range of data engineering problems with simple, chained, stateless [processing steps](../../components/processors/about/). It implements transaction-based resiliency with back pressure, so when connecting to at-least-once sources and sinks it’s able to guarantee at-least-once delivery without needing to persist messages during transit. It’s [simple to deploy](../quickstarts/rpk/), comes with a wide range of [connectors](../../components/about/), and is data agnostic, making it easy to drop into your existing infrastructure. Redpanda Connect has functionality that overlaps with integration frameworks, log aggregators and ETL workflow engines, and can therefore be used to complement these traditional data engineering tools or act as a simpler alternative. ## [](#next-steps)Next steps - [Get started](../quickstarts/rpk/) - [View all connectors](../../components/about/) --- # Page 358: Enterprise Licensing **URL**: https://docs.redpanda.com/redpanda-connect/get-started/licensing.md --- # Enterprise Licensing --- title: Enterprise Licensing latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: licensing page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: licensing.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/get-started/pages/licensing.adoc description: Learn how to purchase and apply an Enterprise Edition license. page-git-created-date: "2024-12-03" page-git-modified-date: "2025-08-28" --- You need a valid Enterprise Edition license to use the following enterprise features: | Feature | Description | | --- | --- | | Allow or deny lists | Limit the Redpanda Connect components that users can run within data pipelines on a Redpanda Connect instance. | | Enterprise connectors | Additional inputs, outputs, and processors available only to enterprise customers. | | FIPS compliance | Run Redpanda Connect using a FIPS-compliant version of rpk, the Redpanda command-line interface (CLI). | | Redpanda Connect configuration service | A configuration block that you can use to send logs and status events to a topic on a Redpanda cluster. | | Secrets management | Retrieve secrets values from a remote system, such as a secret management solution, without setting environment variables. | ## [](#get-an-enterprise-license)Get an enterprise license You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that’s valid for 30 days. ## [](#apply-a-license-key-to-redpanda-connect)Apply a license key to Redpanda Connect When you have downloaded the license key, you can apply it using either the CLI (`rpk`) or Helm on Kubernetes. ### rpk (local or VM) If you have a license key file, either: - Save the license key to `/etc/redpanda/redpanda.license`. - Set the environment variable `REDPANDA_LICENSE_FILEPATH` to point to the file where the license key is stored. If you have a license string, either: - Set the environment variable `REDPANDA_LICENSE` to the value of the full license string. - Run the following command using the full license string: ```bash rpk connect run --redpanda-license ./.yaml ``` Replace the following placeholders: - ``: The full license key string. - ``: The name of your Redpanda Connect configuration file. ### Helm (Kubernetes) To enable enterprise features, you must provide a license key as a Kubernetes Secret and reference it in your deployment configuration. 1. Create the Secret: ```bash kubectl create secret generic redpanda-connect-license \ --from-literal=REDPANDA_LICENSE= \ --namespace ``` 2. Reference the license secret in your `values.yaml` or as a Helm value: ```yaml envFrom: - secretRef: name: redpanda-connect-license ``` This configuration mounts the license key as the `REDPANDA_LICENSE` environment variable, allowing Redpanda Connect to use enterprise features. ## [](#verify-a-license)Verify a license Redpanda Connect checks the license key status at runtime. If the license key is unavailable or has expired, you are blocked from using enterprise connectors. You can view the license key’s expiration date at any time in the Redpanda Connect logs: ```none INFO Running main config from specified file @service=benthos benthos_version=4.50.0 path=connect_2.yaml INFO Successfully loaded Redpanda license @service=benthos expires_at="2025-04-18T15:41:56+01:00" license_org=67XXX license_type="enterprise" ``` ## [](#next-steps)Next steps - [Get started with Redpanda Connect](../quickstarts/). - [Explore enterprise connectors](../../components/about/) to enhance your data pipelines. --- # Page 359: Quickstarts **URL**: https://docs.redpanda.com/redpanda-connect/get-started/quickstarts.md --- # Quickstarts --- title: Quickstarts latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: quickstarts/index page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: quickstarts/index.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/get-started/pages/quickstarts/index.adoc description: Use your preferred tool to spin up a local instance of Redpanda Connect and get hands-on experience building your first pipelines. page-git-created-date: "2024-11-14" page-git-modified-date: "2024-11-14" --- Use your preferred tool to spin up a local instance of Redpanda Connect and get hands-on experience building your first pipelines. - [Get Started with Redpanda Connect using `rpk`](rpk/) Deploy your first pipelines using Redpanda Connect and `rpk`. - [Get Started with the Redpanda Connect Helm Chart](helm-chart/) Deploy Redpanda Connect on Kubernetes using Helm. --- # Page 360: Get Started with the Redpanda Connect Helm Chart **URL**: https://docs.redpanda.com/redpanda-connect/get-started/quickstarts/helm-chart.md --- # Get Started with the Redpanda Connect Helm Chart --- title: Get Started with the Redpanda Connect Helm Chart latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: quickstarts/helm-chart page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: quickstarts/helm-chart.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/get-started/pages/quickstarts/helm-chart.adoc description: Deploy Redpanda Connect on Kubernetes using Helm. page-git-created-date: "2024-11-14" page-git-modified-date: "2025-08-28" --- This guide explains how to deploy and configure Redpanda Connect on Kubernetes using the Helm chart. It covers the available deployment modes and walks you through building your own pipelines. ## [](#what-is-redpanda-connect)What is Redpanda Connect? Redpanda Connect is a powerful stream processor that integrates data across various sources (inputs) and sinks (outputs), enabling seamless data flows between systems. It supports complex data processing tasks such as data enrichment, transformation, filtering, and routing, making it an ideal solution for data pipelines that require high performance and resilience. Redpanda Connect includes Bloblang, a flexible mapping language for processing data with built-in functions for transformations, random data generation, and more. These functions allow for easy customization of data pipelines. Common use cases for Redpanda Connect include: - **Real-time data ingestion**: Ingesting data from various sources into Redpanda or other data platforms. - **Data transformation**: Enriching and transforming data to match business requirements before forwarding it to a destination. - **Data filtering and routing**: Filtering data and routing it to the appropriate sinks based on predefined conditions. ## [](#prerequisites)Prerequisites - A running Kubernetes cluster. - The [`kubectl` CLI](https://kubernetes.io/docs/tasks/tools/) and the [`helm` CLI](https://helm.sh/docs/intro/install/) installed. You can optionally provide an enterprise license to enable enterprise connectors. If you have a license, see [Apply a license key to Redpanda Connect](../../licensing/) before installing. ## [](#install-with-helm)Install with Helm To deploy Redpanda Connect on Kubernetes, use the official Helm chart. To install the chart: ```bash helm repo add redpanda https://charts.redpanda.com (1) helm repo update (2) helm install redpanda-connect redpanda/connect --namespace --create-namespace (3) ``` | 1 | Adds the Redpanda Helm repository. | | --- | --- | | 2 | Updates your local Helm repository cache. | | 3 | Installs Redpanda Connect in the given namespace. You can customize this deployment by configuring values in the Helm chart. | ## [](#deployment-modes)Deployment modes Redpanda Connect can run in two different modes: **standalone mode** and **streams mode**. - **Standalone mode**: Allows you to run a single pipeline at a time, making it suitable for simpler configurations that don’t need to run concurrently. - **Streams mode**: Supports multiple pipelines running simultaneously, with each pipeline managed through a ConfigMap. Streams mode is ideal for complex data processing use cases. ## [](#run-pipelines-in-standalone-mode)Run pipelines in standalone mode In standalone mode, you can configure a single pipeline within the `config` section of your Helm values file. ### [](#hello-world)Hello world In this example, you’ll produce a simple message and convert it to uppercase using a Bloblang method. 1. Create a `pipeline.yaml` file. This file will be used to override the default Helm chart values for Redpanda Connect. 2. Add the following configuration to `pipeline.yaml`: ```yaml config: input: generate: mapping: | root.message = "Hello, Redpanda Connect!" (1) count: 1 (2) pipeline: processors: - mapping: | root.message = this.message.uppercase() (3) output: stdout: {} (4) ``` | 1 | Use Bloblang to create a JSON object with the message "Hello, Redpanda Connect!" stored in root.message. | | --- | --- | | 2 | Generate the message only once. | | 3 | Convert the message field in the input to uppercase using the uppercase() method in Bloblang. | | 4 | Output the processed data to stdout, making it viewable in logs. | 3. Deploy the pipeline: ```bash helm upgrade --install redpanda-connect redpanda/connect --namespace --values pipeline.yaml ``` 4. Check the logs: ```bash export POD_NAME=$(kubectl get pods --namespace -l "app.kubernetes.io/name=redpanda-connect,app.kubernetes.io/instance=redpanda-connect" -o jsonpath="{.items[0].metadata.name}") kubectl logs $POD_NAME --namespace ``` You should see the message converted to uppercase in the output: ```json {"message":"HELLO, REDPANDA CONNECT!"} ``` 5. Check the Pod’s status: ```bash kubectl get pods --namespace -l app.kubernetes.io/name=redpanda-connect --watch ``` The Pod enters a `CrashLoopBackOff` state because containers are expected to run continuously. When Redpanda Connect finishes processing the pipeline, the Pod exits, causing Kubernetes to restart it repeatedly. To prevent this status, you can configure Redpanda Connect to continue processing data indefinitely. ### [](#produce-continuous-data)Produce continuous data To produce data continuously, you can set `input.generate.count` to `0`. 1. Update the `pipeline.yaml` file to produce a message every second, indefinitely: ```yaml config: input: generate: interval: 1s count: 0 # Setting count to 0 ensures it generates data indefinitely. mapping: | root.message = "Hello, Redpanda Connect!" pipeline: processors: - mapping: | root.message = this.uppercase() output: stdout: {} ``` 2. Deploy the updated configuration: ```bash helm upgrade --install redpanda-connect redpanda/connect --namespace --values pipeline.yaml ``` 3. Watch the logs: ```bash export POD_NAME=$(kubectl get pods --namespace -l "app.kubernetes.io/name=redpanda-connect,app.kubernetes.io/instance=redpanda-connect" -o jsonpath="{.items[0].metadata.name}") kubectl logs $POD_NAME --namespace -f ``` You should see in the logs that Redpanda Connect is producing the same message every second and its being converted to uppercase: ```json {"message": "HELLO, REDPANDA CONNECT!"} {"message": "HELLO, REDPANDA CONNECT!"} {"message": "HELLO, REDPANDA CONNECT!"} ``` 4. Check the Pod’s status: ```bash kubectl get pods --namespace -l app.kubernetes.io/name=redpanda-connect --watch ``` The Pod should now be running without entering a `CrashLoopBackOff` state, as the `generate` input continuously feeds new data to the pipeline, preventing it from terminating. ### [](#simulate-realistic-data-streams)Simulate realistic data streams To make the output more realistic, use some Bloblang functions to generate varied data such as random names and emails. 1. Update the `pipeline.yaml` file to generate some realistic user data. ```yaml config: input: generate: interval: 1s count: 0 mapping: | # Store the generated names in variables let first_name = fake("first_name") let last_name = fake("last_name") # Build the message root.user_id = counter() root.name = ($first_name + " " + $last_name) root.timestamp = now() pipeline: processors: - mapping: | root.name = this.name.uppercase() output: stdout: {} ``` This configuration generates a JSON object with: - `user_id`: A unique identifier for each record, generated using the [`counter()`](../../../guides/bloblang/functions/#counter) function. - `name`: A randomly generated first and last name, using the [`fake()`](../../../guides/bloblang/functions/#fake) function. The first and last names are stored in variables and referenced using the `$` syntax. - `timestamp`: The current timestamp at the time of generation, using the [`now()`](../../../guides/bloblang/functions/#now) function. 2. Deploy the updated configuration: ```bash helm upgrade --install redpanda-connect redpanda/connect --namespace --values pipeline.yaml ``` 3. Watch the logs: ```bash export POD_NAME=$(kubectl get pods --namespace -l "app.kubernetes.io/name=redpanda-connect,app.kubernetes.io/instance=redpanda-connect" -o jsonpath="{.items[0].metadata.name}") kubectl logs $POD_NAME --namespace -f ``` You should see logs showing JSON objects similar to the following, with names in uppercase: ```json {"name":"ZOIE SIPES"} {"name":"LORENA KERTZMANN"} {"name":"DALLAS BOYER"} {"name":"LOUIE WILDERMAN"} {"name":"EMILIA KOEPP"} {"name":"KALEIGH PACOCHA"} ``` ### [](#process-data-from-a-file-input)Process data from a file input To configure a pipeline that reads data from a file, first store the data in a ConfigMap. This ConfigMap will be mounted into the Redpanda Connect Pod, allowing it to read the file directly. 1. Create a ConfigMap to provide the input data that Redpanda Connect will read. This example ConfigMap contains a JSON object with example user data: ```bash kubectl create configmap connect-input --from-literal=input-data='{"name": "Redpanda Connect", "email": "rp.connect@example.com"}' --namespace ``` This ConfigMap will act as the source for the file-based input in Redpanda Connect, allowing the pipeline to read and process this structured JSON data. 2. Update the `pipeline.yaml` file to read data from the file mounted by the ConfigMap: `pipeline.yaml` ```yaml extraVolumes: - name: input-config configMap: name: connect-input extraVolumeMounts: - name: input-config mountPath: /input (1) subPath: input-data config: input: file: paths: - "/input" (1) pipeline: processors: - mapping: | root.name = this.name.uppercase() output: stdout: {} ``` | 1 | For the input, use the contents of the file at the path where the ConfigMap data is mounted. | | --- | --- | 3. Deploy the pipeline: ```bash helm upgrade --install redpanda-connect redpanda/connect --namespace --values pipeline.yaml ``` 4. Check the logs: ```bash export POD_NAME=$(kubectl get pods --namespace -l "app.kubernetes.io/name=redpanda-connect,app.kubernetes.io/instance=redpanda-connect" -o jsonpath="{.items[0].metadata.name}") kubectl logs $POD_NAME --namespace ``` You should see the username converted to uppercase in the output: ```json {"name":"REDPANDA CONNECT"} ``` ## [](#run-multiple-pipelines-in-streams-mode)Run multiple pipelines in streams mode In streams mode, each pipeline, defined in separate YAML files, runs simultaneously, making this mode ideal for high-throughput applications. All the YAML files must be bundled together into a ConfigMap that you can pass to Redpanda Connect. 1. Define your pipeline configurations in the following separate YAML files: `woof.yaml` ```yaml input: generate: mapping: root = "woof" # Generates a message with the word "woof" at regular intervals. interval: 5s count: 0 output: stdout: codec: lines # Outputs each message as a new line in stdout. ``` `meow.yaml` ```yaml input: generate: mapping: root = "meow" # Generates a message with the word "meow" at regular intervals. interval: 2s count: 0 output: stdout: codec: lines # Outputs each message as a new line in stdout. ``` 2. Bundle the configuration files into a ConfigMap, which Redpanda Connect will reference: ```bash kubectl create configmap connect-streams --from-file=woof.yaml --from-file=meow.yaml --namespace ``` 3. Configure Redpanda Connect in streams mode and specify the name of the ConfigMap to use: `connect.yaml` ```yaml streams: enabled: true (1) streamsConfigMap: "connect-streams" (2) ``` | 1 | Enable streams mode in Redpanda Connect. | | --- | --- | | 2 | Use the given ConfigMap as the pipeline configuration. | 4. Deploy the chart: ```bash helm upgrade --install redpanda-connect redpanda/connect --namespace --values connect.yaml ``` 5. Watch the logs: ```bash export POD_NAME=$(kubectl get pods --namespace -l "app.kubernetes.io/name=redpanda-connect,app.kubernetes.io/instance=redpanda-connect" -o jsonpath="{.items[0].metadata.name}") kubectl logs $POD_NAME --namespace -f ``` You should see logs showing a combination of outputs from both pipelines: woof meow meow meow woof meow meow ### [](#update-the-pipeline-in-streams-mode)Update the pipeline in streams mode To update a pipeline in streams mode: 1. Modify one of the configuration files locally. `woof.yaml` ```yaml # Updated woof.yaml input: generate: mapping: root = "bark" # Updated to generate a message with the word "bark" instead of "woof." interval: 5s count: 0 output: stdout: codec: lines ``` 2. Update the ConfigMap with the modified file: ```bash kubectl create configmap connect-streams --from-file=woof.yaml --from-file=meow.yaml --namespace --dry-run=client -o yaml | kubectl apply -f - ``` 3. Restart the Deployment: ```bash kubectl rollout restart deployment/redpanda-connect --namespace ``` ### [](#global-configuration)Global configuration When deploying Redpanda Connect in streams mode, you can configure global tracing, logging, and HTTP settings to apply across all pipelines. Specify these in your `values.yaml` overrides under the `metrics`, `logger`, and `tracing` sections. ```yaml metrics: prometheus: {} # Enable Prometheus metrics collection. tracing: openTelemetry: http: [] # Configure OpenTelemetry HTTP tracing. grpc: [] tags: {} logger: level: INFO # Set logging level (e.g., INFO, DEBUG). static_fields: '@service': redpanda-connect # Add static fields to logs for better traceability. ``` ## [](#access-the-http-server-on-redpanda-connect)Access the HTTP server on Redpanda Connect To manage and monitor Redpanda Connect, you can use its HTTP server, which provides useful endpoints for version checking, pipeline management, and more. By default, Redpanda Connect exposes this server using a Kubernetes ClusterIP Service, accessible only within the cluster. 1. Forward the ports of the ClusterIP Service to your local device: ```bash kubectl port-forward svc/redpanda-connect 8080:80 --namespace ``` 2. Access the HTTP server locally. For example, to check the Redpanda Connect version, run: ```bash curl http://localhost:8080/version ``` Example output: ```json { "version": "v4.38.0", "built": "2024-10-17T09:27:42Z" } ``` You can also configure external access using a LoadBalancer Service or an Ingress. See the [Helm values](https://github.com/redpanda-data/helm-charts/blob/main/charts/connect/values.yaml#L79C1-L107C31) for more details. ## [](#uninstall-redpanda-connect)Uninstall Redpanda Connect To remove Redpanda Connect and all related resources from your Kubernetes cluster, use the [`helm uninstall`](https://helm.sh/docs/helm/helm_uninstall/) command to uninstall the chart: ```bash helm uninstall redpanda-connect --namespace ``` This command deletes all resources created by the Helm chart, including Deployments and Services. Uninstalling the chart does not delete the ConfigMaps that you manually created outside of the Helm chart. To delete these ConfigMaps, do the following: ```bash kubectl delete configmap connect-streams connect-input --namespace ``` ## [](#next-steps)Next steps - Learn more about [Bloblang](../../../guides/bloblang/walkthrough/), the mapping language for processing data in Redpanda Connect. - Try more hands-on examples with one of the [Cookbooks](../../../cookbooks/). ## [](#suggested-reading)Suggested reading - [Install or Upgrade with the Helm Chart](../../../install/helm-chart/) - [Streams mode](../../../guides/streams_mode/about/) - [Inputs](../../../components/inputs/about/) - [Processors](../../../components/processors/about/) - [Outputs](../../../components/outputs/about/) - [HTTP server](../../../components/http/about/) - [Helm values](https://github.com/redpanda-data/helm-charts/blob/main/charts/connect/values.yaml) --- # Page 361: Get Started with Redpanda Connect using rpk **URL**: https://docs.redpanda.com/redpanda-connect/get-started/quickstarts/rpk.md --- # Get Started with Redpanda Connect using `rpk` --- title: Get Started with Redpanda Connect using rpk latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: quickstarts/rpk page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: quickstarts/rpk.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/get-started/pages/quickstarts/rpk.adoc description: Deploy your first pipelines using Redpanda Connect and rpk. page-git-created-date: "2024-11-14" page-git-modified-date: "2026-03-31" --- This guide explains how to get started with Redpanda Connect using `rpk`, the Redpanda command-line interface (CLI). You can also install and run `rpk` in [FIPS compliance mode](#fips-compliance). ## [](#install)Install The `rpk` CLI allows you to create and manage data pipelines with Redpanda Connect as well as [interact with Redpanda clusters](../../../../current/get-started/rpk-install/). > 📝 **NOTE** > > The `rpk` CLI also includes an `rpk connect` plugin, which manages installations and upgrades of Redpanda Connect. This plugin is automatically installed when you first run `rpk connect` commands, unless you run `rpk connect --version`, which prompts you to install the plugin. Also interacting with a Redpanda cluster? If you want to use `rpk` to also communicate with a Redpanda cluster, ensure the version of `rpk` that you install matches the version of Redpanda running in your cluster. ### [](#linux)Linux To install, or update to, the latest version of `rpk` for Linux, run: #### amd64 ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-amd64.zip -d ~/.local/bin/ ``` #### arm64 ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-arm64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-arm64.zip -d ~/.local/bin/ ``` > 💡 **TIP** > > You can use `rpk` on Windows only with [WSL](https://learn.microsoft.com/windows/wsl/install). However, commands that require Redpanda to be installed on your machine are not supported, such as [`rpk container`](../../../../current/reference/rpk/rpk-container/rpk-container/) commands, [`rpk iotune`](../../../../current/reference/rpk/rpk-iotune/), and [`rpk redpanda`](../../../../current/reference/rpk/rpk-redpanda/rpk-redpanda/) commands. To install, or update to, a version other than the latest, run: #### amd64 ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/download/v/rpk-linux-amd64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-amd64.zip -d ~/.local/bin/ ``` #### arm64 ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/download/v/rpk-linux-arm64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-arm64.zip -d ~/.local/bin/ ``` #### [](#fips-compliance)FIPS compliance > 📝 **NOTE** > > This feature requires an [enterprise license](../../licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that’s valid for 30 days. To install `rpk` to run the latest version of Redpanda Connect in FIPS-compliant mode, you must install the `redpanda-rpk-fips` and `redpanda-connect-fips` packages. Both packages are built using the [Microsoft GoLang compiler](https://github.com/microsoft/go) and the [Microsoft’s Go Crypto OpenSSL package](https://github.com/microsoft/go-crypto-openssl), which uses the FIPS-approved version of OpenSSL. > ❗ **IMPORTANT** > > The packages for FIPS-compliant `rpk` (`redpanda-rpk-fips`) and Redpanda `rpk` (`redpanda-rpk`) are mutually exclusive, and so cannot be installed in the same environment. ##### RHEL 1. To make sure your distribution is up to date, run: ```bash sudo dnf upgrade ``` 2. Add `redpanda` to your `dnf` list of repositories. ```bash curl -1sLf 'https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.rpm.sh' | \ sudo -E bash ``` 3. Install Redpanda packages for FIPS compliance. ```bash sudo dnf install -y redpanda-rpk-fips redpanda-connect-fips ``` 4. Verify your installation. ```bash rpk connect --version ``` To keep up-to-date with Redpanda Connect releases, run the following command: ```bash sudo dnf update ``` ##### Debian/Ubuntu 1. To make sure your distribution is up to date, run: ```bash sudo apt upgrade ``` 2. Add `redpanda` to your `apt` list of repositories. ```bash curl -1sLf 'https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.deb.sh' | sudo -E bash ``` 3. Install Redpanda packages for FIPS compliance. ```bash sudo apt install -y redpanda-rpk-fips redpanda-connect-fips ``` 4. Verify your installation. ```bash rpk connect --version ``` To keep up-to-date with the Redpanda Connect releases, run the following command: ```bash sudo apt update ``` ### [](#macos)MacOS #### Homebrew 1. If you don’t have Homebrew installed, [install it](https://brew.sh/). 2. To install or update `rpk`, run: ```bash brew install redpanda-data/tap/redpanda ``` #### Manual Download To install or update `rpk` through a manual download, choose the option for your system architecture. For example, if you have an M1 or newer chip, select **Apple Silicon**. ##### Intel macOS To install, or update to, the latest version of `rpk` for Intel macOS, run: ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-darwin-amd64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-darwin-amd64.zip -d ~/.local/bin/ ``` To install, or update to, a version other than the latest, run: ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/download/v/rpk-darwin-amd64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-darwin-amd64.zip -d ~/.local/bin/ ``` ##### Apple Silicon To install, or update to, the latest version of `rpk` for Apple Silicon, run: ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-darwin-arm64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-darwin-arm64.zip -d ~/.local/bin/ ``` To install, or update to, a version other than the latest, run: ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/download/v/rpk-darwin-arm64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-darwin-arm64.zip -d ~/.local/bin/ ``` ### [](#docker)Docker To use `rpk` in Docker, you can use the Redpanda Docker image, which includes both Redpanda and `rpk` as part of the installation. ```bash docker pull docker.redpanda.com/redpandadata/connect ``` ## [](#run-a-pipeline)Run a pipeline A Redpanda Connect stream pipeline is configured with a single [config file](../../../configuration/about/), you can generate a fresh one with: ```bash rpk connect create > connect.yaml ``` > 📝 **NOTE** > > This command may take a few seconds to run. If this is the first `rpk connect` command you have run, the `rpk connect plugin` is automatically installed. For Docker installations: ```bash docker run --rm docker.redpanda.com/redpandadata/connect create > ./connect.yaml ``` The main sections that make up a config are `input`, `pipeline` and `output`. When you generate a fresh config it’ll simply pipe `stdin` to `stdout` like this: ```yaml input: stdin: {} pipeline: processors: [] output: stdout: {} ``` Eventually we’ll want to configure a more useful [input](../../../components/inputs/about/) and [output](../../../components/outputs/about/), but for now this is useful for quickly testing processors. You can execute this config with: ```bash rpk connect run connect.yaml ``` For Docker installations: ```bash docker run --rm -it -v $(pwd)/connect.yaml:/connect.yaml docker.redpanda.com/redpandadata/connect run ``` Anything you write to stdin will get written unchanged to stdout! Resist the temptation to play with this for hours, there’s more stuff to try out. Next, let’s add some processing steps in order to mutate messages. The most powerful one is the [`mapping` processor](../../../components/processors/mapping/) which allows us to perform mappings, let’s add a mapping to uppercase our messages: ```yaml input: stdin: {} pipeline: processors: - mapping: root = content().uppercase() output: stdout: {} ``` Now your messages should come out in all caps. You can add as many [processing steps](../../../components/processors/about/) as you like, and since processors are what make Redpanda Connect powerful they are worth experimenting with. Let’s create a more advanced pipeline that works with JSON documents: ```yaml input: stdin: {} pipeline: processors: - sleep: duration: 500ms - mapping: | root.doc = this root.first_name = this.names.index(0).uppercase() root.last_name = this.names.index(-1).hash("sha256").encode("base64") output: stdout: {} ``` First, we sleep for 500 milliseconds just to keep the suspense going. Next, we restructure our input JSON document by nesting it within a field `doc`, we map the upper-cased first element of `names` to a new field `first_name`. Finally, we map the hashed and base64 encoded value of the last element of `names` to a new field `last_name`. Try running that config with some sample documents: ```bash echo '{"id":"1","names":["celine","dion"]} {"id":"2","names":["chad","robert","kroeger"]}' | rpk connect run connect.yaml ``` For Docker installations: ```bash echo '{"id":"1","names":["celine","dion"]} {"id":"2","names":["chad","robert","kroeger"]}' | docker run --rm -i -v $(pwd)/connect.yaml:/connect.yaml docker.redpanda.com/redpandadata/connect run ``` You should see this output in the logs: ```json {"doc":{"id":"1","names":["celine","dion"]},"first_name":"CELINE","last_name":"1VvPgCW9sityz5XAMGdI2BTA7/44Wb3cANKxqhiCo50="} {"doc":{"id":"2","names":["chad","robert","kroeger"]},"first_name":"CHAD","last_name":"uXXg5wCKPjpyj/qbivPbD9H9CZ5DH/F0Q1Twytnt2hQ="} ``` See also: - [Bloblang Walkthrough](../../../guides/bloblang/walkthrough/) - [Inputs](../../../components/inputs/about/) - [Processors](../../../components/processors/about/) - [Outputs](../../../components/outputs/about/) - [Monitoring](../../../guides/monitoring/) - [Cookbooks](../../../cookbooks/) - [More about configuration](../../../configuration/about/) --- # Page 362: Redpanda Connect Versioning **URL**: https://docs.redpanda.com/redpanda-connect/get-started/versioning.md --- # Redpanda Connect Versioning --- title: Redpanda Connect Versioning latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: versioning page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: versioning.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/get-started/pages/versioning.adoc description: Learn how Redpanda Connect implements versioning. page-git-created-date: "2025-09-25" page-git-modified-date: "2025-09-25" --- Redpanda Connect uses semantic versioning. Version numbers follow the format `major.minor.patch`: - `major`: This version may include breaking changes. - `minor`: This version only includes backward-compatible new features or enhancements. - `patch`: This version includes backward-compatible bug fixes. For example, version 4.35.1 is the first patch release of bug fixes following the minor release of product enhancements in 4.35.0. Breaking changes would not be released until 5.0.0. ## [](#next-steps)Next steps To benefit from Redpanda Connect’s new features and enhancements, [install the latest version](../../install/). Learn more about the [versioning of other Redpanda Self-Managed products](../../../current/upgrade/rolling-upgrade/). --- # Page 363: What’s New in Redpanda Connect **URL**: https://docs.redpanda.com/redpanda-connect/get-started/whats-new.md --- # What’s New in Redpanda Connect --- title: What’s New in Redpanda Connect latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: whats-new page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: whats-new.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/get-started/pages/whats-new.adoc description: Summary of new features in Redpanda Connect. page-git-created-date: "2024-11-14" page-git-modified-date: "2026-04-10" --- This topic includes new content added from version 4.29.0 onwards. For a full list of product updates, and details of updates prior to version 4.29.0, see the [Redpanda Connect changelog](https://github.com/redpanda-data/connect/blob/main/CHANGELOG.md). For new Cloud-specific features, see [What’s New in Redpanda Cloud](../../../redpanda-cloud/get-started/whats-new-cloud/). ## [](#version-4-87-0)Version 4.87.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.87.0). ### [](#new-field-support)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | checkpoint_cache_connection_string | An optional connection string for a remote Microsoft SQL Server to use for the checkpoint cache. When set, the checkpoint cache table is created on this remote server instead of the source database. | Inputs:microsoft_sql_server_cdc | | wallet_path | Path to the Oracle Wallet directory. When set, SSL is enabled automatically. | Inputs:oracledb_cdc | | wallet_password | Password for the ewallet.p12 PKCS#12 wallet file. Only required when the wallet directory contains ewallet.p12 rather than cwallet.sso. | Inputs:oracledb_cdc | | message_properties_message_id | Set the message-id property on outgoing AMQP messages. The value is auto-detected as UUID, uint64, or string. | Outputs:amqp_1 | | message_properties_correlation_id | Set the correlation-id property on outgoing AMQP messages. The value is auto-detected as UUID, uint64, or string. | Outputs:amqp_1 | | message_properties_subject | Set the subject property on outgoing AMQP messages. This field supports Bloblang interpolation. | Outputs:amqp_1 | | message_properties_reply_to | Set the reply-to property on outgoing AMQP messages. This field supports Bloblang interpolation. | Outputs:amqp_1 | | message_properties_group_id | Set the group-id property on outgoing AMQP messages. This field supports Bloblang interpolation. | Outputs:amqp_1 | | message_properties_group_sequence | Set the group-sequence property on outgoing AMQP messages. Must be a valid uint32 value. | Outputs:amqp_1 | | message_properties_reply_to_group_id | Set the reply-to-group-id property on outgoing AMQP messages. This field supports Bloblang interpolation. | Outputs:amqp_1 | | message_properties_user_id | Set the user-id property on outgoing AMQP messages. This field supports Bloblang interpolation. | Outputs:amqp_1 | | message_properties_content_type | Set the content-type property on outgoing AMQP messages. This field supports Bloblang interpolation. | Outputs:amqp_1 | | message_properties_content_encoding | Set the content-encoding property on outgoing AMQP messages. This field supports Bloblang interpolation. | Outputs:amqp_1 | ## [](#version-4-86-0)Version 4.86.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.86.0). ### [](#component-updates)Component updates This release adds the following new components: | Component | Type | Status | Description | | --- | --- | --- | --- | | string_split | Processors | stable | Splits a string by a delimiter into an array. Generally, using bloblang’s split method is preferred. In some high performance use cases this processor can be faster than the equivalent bloblang if there is no additional logic. | ## [](#version-4-85-0)Version 4.85.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.85.0). ### [](#bloblang-updates)Bloblang updates This release adds the following new Bloblang capabilities: - Methods: - [`escape_url_path`](../../guides/bloblang/methods/#escape_url_path) Escapes a string so it can be safely placed inside a URL path segment, replacing special characters with percent-encoded sequences. - [`parse_logfmt`](../../guides/bloblang/methods/#parse_logfmt) Parses a logfmt-encoded string into an object of key-value pairs. - [`unescape_url_path`](../../guides/bloblang/methods/#unescape_url_path) Unescapes a URL path segment, converting percent-encoded sequences back to their original characters. ### [](#component-updates-2)Component updates This release adds the following new components: | Component | Type | Description | | --- | --- | --- | | oracledb_cdc | Inputs | Streams changes from an Oracle database for Change Data Capture (CDC). | | salesforce_sink | Outputs | Consumes batches of messages and writes them to Salesforce. | | salesforce | Processors | Fetches data from Salesforce based on input messages. | ### [](#new-field-support-2)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | client_name | Set the client name for the Redis connection. | Caches:redisInputs:redis_listredis_pubsubredis_scanredis_streamsOutputs:redis_hashredis_listredis_pubsubredis_streamsProcessors:redisredis_scriptRate-limits:redis | | tables | List of table names to stream from. For single table mode, provide one table. | Inputs:aws_dynamodb_cdc | | table_discovery_mode | Table discovery mode. | Inputs:aws_dynamodb_cdc | | table_tag_filter | Multi-tag filter: 'key1:v1,v2;key2:v3,v4'. Matches tables with (key1=v1 OR key1=v2) AND (key2=v3 OR key2=v4). | Inputs:aws_dynamodb_cdc | | table_discovery_interval | Interval for rescanning and discovering new tables when using tag or includelist mode. Set to 0 to disable periodic rescanning. | Inputs:aws_dynamodb_cdc | | snapshot_mode | Snapshot behavior. | Inputs:aws_dynamodb_cdc | | snapshot_segments | Number of parallel scan segments (1-10). Higher parallelism scans faster but consumes more RCUs. | Inputs:aws_dynamodb_cdc | | snapshot_batch_size | Records per scan request during snapshot. Maximum 1000. | Inputs:aws_dynamodb_cdc | | snapshot_throttle | Minimum time between scan requests per segment. Use this to limit RCU consumption during snapshot. | Inputs:aws_dynamodb_cdc | | snapshot_deduplicate | Deduplicate records that appear in both snapshot and CDC stream. Requires buffering CDC events during snapshot. | Inputs:aws_dynamodb_cdc | | snapshot_buffer_size | Maximum CDC events to buffer for deduplication (approximately 100 bytes per entry). If exceeded, deduplication is disabled and duplicates may be emitted. | Inputs:aws_dynamodb_cdc | | create_stream | Whether to automatically create the stream if it doesn’t exist (requires the stream field to be set). | Inputs:nats_jetstream | | extract_tracing_map | A Bloblang mapping that attempts to extract an object containing tracing propagation information, which will then be used as the root tracing span for the message. The specification of the extracted fields must match the format used by the service wide tracer. | Inputs:redpanda | | id | The entry ID for the stream message. Allows function interpolations. | Outputs:redis_streams | | inject_tracing_map | A Bloblang mapping used to inject an object containing tracing propagation information into outbound messages. The specification of the injected fields will match the format used by the service wide tracer. | Outputs:redpanda | | sync_topic_interval | How often to synchronize topics from the source cluster to the destination. This creates destination topics for any new source topics, including empty topics with no message flow. | Outputs:redpanda_migrator | | commit_backoff | Control how frequently Snowflake is polled to check if data has been committed. | Outputs:snowflake_streaming | | timeout | HTTP request timeout. | Processors:jira | | tls | Custom TLS settings can be used to override system defaults. | Processors:jira | | proxy_url | HTTP proxy URL. Empty string disables proxying. | Processors:jira | | disable_http2 | Disable HTTP/2 and force HTTP/1.1. | Processors:jira | | tps_limit | Rate limit in requests per second. 0 disables rate limiting. | Processors:jira | | tps_burst | Maximum burst size for rate limiting. | Processors:jira | | backoff | Adaptive backoff configuration for 429 (Too Many Requests) responses. Always active. | Processors:jira | | tcp | TCP socket configuration. | Processors:jira | | http | HTTP transport settings controlling connection pooling, timeouts, and HTTP/2. | Processors:jira | | access_log_level | Log level for HTTP request/response logging. Empty disables logging. | Processors:jira | | access_log_body_limit | Maximum bytes of request/response body to include in logs. 0 to skip body logging. | Processors:jira | | schema_metadata | When set, the processor reads a schema in benthos common schema format from this metadata key on each message, converts it to the format specified by format, registers it with the schema registry under the configured subject, and encodes the message. When empty (the default), the processor pulls the latest schema from the registry instead. | Processors:schema_registry_encode | | format | The encoding format to use when converting a common schema from metadata. | Processors:schema_registry_encode | | normalize | Whether to normalize the schema before registering with the schema registry (schema_metadata mode only). | Processors:schema_registry_encode | | avro | Configuration for Avro encoding. | Processors:schema_registry_encode | | continue | Indicates whether, if this case passes for a message, the next case should also be tested. | Processors:switch | ### [](#deprecations)Deprecations The following fields are now deprecated: | Field | Description | Affected components | | --- | --- | --- | | commit_timeout | Use max_elapsed_time instead. | Outputs:snowflake_streaming | | avro_raw_json | Use avro.raw_json instead. | Processors:schema_registry_encode | ### [](#default-value-changes)Default value changes This release includes the following default value changes: | Field | Old default | New default | Description | Affected components | | --- | --- | --- | --- | --- | | target_address | none | | The target address to write to. | Outputs:amqp_1 | | commit_timeout | 60s | | Use max_elapsed_time instead. | Outputs:snowflake_streaming | ### [](#removed-components)Removed components The following deprecated components have been removed: - `legacy_redpanda_migrator` input and output - `legacy_redpanda_migrator_offsets` input and output - `redpanda_migrator_bundle` input and output Use the unified [`redpanda_migrator`](../../components/inputs/redpanda_migrator/) input and output instead. ## [](#version-4-81-0)Version 4.81.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.81.0). ### [](#component-updates-3)Component updates This release adds the following new components: | Component | Type | Description | | --- | --- | --- | | aws_cloudwatch_logs | Input | Consumes log events from AWS CloudWatch Logs. | ### [](#new-field-support-3)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | timeout_seconds | The timeout in seconds, for querying the TigerBeetle cluster. | Inputs:tigerbeetle_cdc | | subject | An optional subject to set for messages. | Outputs:aws_sns | | shared_drives | Whether or not to include shared drives. | Processors:google_drive_downloadgoogle_drive_search | ## [](#version-4-80-0)Version 4.80.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.80.0). ### [](#component-updates-4)Component updates This release adds the following new components: | Component | Type | Description | | --- | --- | --- | | iceberg | Output | Write data to Apache Iceberg tables via REST catalog. | ## [](#version-4-79-0)Version 4.79.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.79.0). ### [](#component-updates-5)Component updates This release adds the following new components: | Component | Type | Description | | --- | --- | --- | | aws_dynamodb_cdc | Inputs | Consumes records from DynamoDB Streams with automatic checkpointing and shard management. | ### [](#new-field-support-4)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | encoding | Encoding format for messages in the batch. Options: 'protobuf' or 'json'. | Inputs:otlp_grpcotlp_http | | schema_registry | Optional Schema Registry configuration for adding Schema Registry wire format headers to messages. | Inputs:otlp_grpcotlp_http | | message_format | The format to expect incoming messages from the rest of the pipeline. | Outputs:snowflake_streaming | | timestamp_format | The format to parse string values for TIMESTAMP, TIMESTAMP_LTZ and TIMESTAMP_NTZ columns. Should be a layout for time.Parse in Go. | Outputs:snowflake_streaming | ## [](#version-4-78-0)Version 4.78.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.78.0). ### [](#new-field-support-5)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | tcp | TCP listener configuration for the HTTP server. | Inputs:http_server | ## [](#version-4-77-0)Version 4.77.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.77.0). ### [](#component-updates-6)Component updates This release adds the following new components: | Component | Type | Description | | --- | --- | --- | | elasticsearch_v9 | Outputs | Publish messages into an Elasticsearch index. | ## [](#version-4-76-1)Version 4.76.1 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.76.1). ## [](#version-4-76-0)Version 4.76.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.76.0). ### [](#component-updates-7)Component updates This release adds the following new components: > 📝 **NOTE** > > Requires a cgo-enabled binary. See the [installation guides](../../install/) for details. ### [](#new-field-support-6)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | tags | Global tags added to each metric. | Metrics:statsd | ## [](#version-4-75-0)Version 4.75.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.75.0). ### [](#new-field-support-7)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | batching | Optional explicit batching policy for the output. Note that when batches are formed at the input level they can be expanded by this policy, but not contracted. | Outputs:redpanda | ## [](#version-4-74-0)Version 4.74.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.74.0). ## [](#version-4-73-0)Version 4.73.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.73.0). ### [](#new-field-support-8)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | tcp | TCP listener socket configuration. | Inputs:socket_server | | ttl | An optional TTL to set for items. | Outputs:couchbaseProcessors:couchbase | | schema_registry | Schema registry information to publish schemas for tracing data along with the data. | Tracers:redpanda | ## [](#version-4-72-0)Version 4.72.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.72.0). ## [](#version-4-71-0)Version 4.71.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.71.0). ### [](#component-updates-8)Component updates This release adds the following new components: | Component | Type | Description | | --- | --- | --- | | redpanda | Tracers | - | ### [](#new-field-support-9)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | aws | AWS IAM authentication configuration for PostgreSQL instances. When enabled, IAM credentials are used to generate temporary authentication tokens instead of a static password. | Inputs:pg_streampostgres_cdc | | offset_header | Header name to add to migrated records containing the source offset for exact consumer group migration. If empty, no offset header is added and exact offset translation is disabled. | Outputs:redpanda_migrator | | max_in_flight | Maximum number of batches to have in flight at any given time. For optimal throughput, set this to the total number of partitions being copied in parallel (up to all partitions in the cluster). | Outputs:redpanda_migrator | ## [](#version-4-70-0)Version 4.70.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.70.0). ### [](#bloblang-updates-2)Bloblang updates This release adds the following new Bloblang capabilities: - Functions: - [`bytes`](../../guides/bloblang/functions/#bytes): Create a new byte array that is zero initialized. - Methods: - [`repeat`](../../guides/bloblang/methods/#repeat): Creates a new array by repeating the elements of the input array a specified number of times. ### [](#new-field-support-10)New field support This release adds support for the following new fields: | Field | Description | Affected components | | --- | --- | --- | | tcp | Configure TCP socket-level settings to optimize network performance and reliability.Most users should keep the default values. Only modify these settings if you’re experiencing connection stability issues or have specific network requirements. | Config:redpandaCaches:aws_dynamodbaws_s3redpandaInputs:aws_kinesisaws_s3aws_sqskafka_franzlegacy_redpanda_migratorlegacy_redpanda_migrator_offsetsredpandaredpanda_migratorOutputs:aws_dynamodbaws_kinesisaws_kinesis_firehoseaws_s3aws_snsaws_sqskafka_franzlegacy_redpanda_migratorlegacy_redpanda_migrator_offsetsredpandaredpanda_migratorProcessors:aws_bedrock_chataws_bedrock_embeddingsaws_dynamodb_partiqlaws_lambdaMetrics:aws_cloudwatch | | regexp_topics_include | A list of regular expression patterns for matching topics to consume from. When specified, the client will periodically refresh the list of matching topics based on the metadata_max_age interval. | Inputs:kafka_franzlegacy_redpanda_migratorredpandaredpanda_commonredpanda_migrator | | regexp_topics_exclude | A list of regular expression patterns for excluding topics. Topics matching any of these patterns will be excluded from consumption, even if they match include patterns. | Inputs:kafka_franzlegacy_redpanda_migratorredpandaredpanda_commonredpanda_migrator | | max_parallel_snapshot_tables | The number of tables that will be processed in parallel during the snapshot processing stage. | Inputs:microsoft_sql_server_cdc | | tls | Custom TLS settings to override system defaults. | Inputs:pg_streampostgres_cdc | | persistent | If set to true, the message is marked as persistent, ensuring it is stored durably and not lost if an intermediary (such as a broker) restarts. By default, messages are not durable. | Outputs:amqp_1 | | target_capabilities | Lists the extension capabilities the sender desires from the target, such as support for queues, topics, durability, sharing, or temporary destinations. | Outputs:amqp_1 | | message_properties_to | The node that is the intended destination of the message, which may differ from the node currently receiving the transfer. | Outputs:amqp_1 | | provenance_header | Header name to add to migrated records indicating their source cluster. When set, each migrated message receives a header with this name containing the source cluster’s seed broker addresses, enabling downstream systems to track message origins for auditing, debugging, or multi-cluster orchestration workflows. | Outputs:redpanda_migrator | ### [](#deprecations-2)Deprecations The following fields are now deprecated: | Field | Description | Affected components | | --- | --- | --- | | regexp_topics | This boolean field is deprecated in favor of regexp_topics_include and regexp_topics_exclude, which provide more explicit control over topic pattern matching. To migrate, replace regexp_topics: true with regexp_topics_include containing your topic patterns, and optionally add regexp_topics_exclude to filter out specific topics. For example, if you had topics: ["events_."] with regexp_topics: true, change to regexp_topics_include: ["events_."]. | Inputs:kafka_franzlegacy_redpanda_migratorredpandaredpanda_commonredpanda_migrator | ### [](#default-value-changes-2)Default value changes This release includes the following default value changes: | Field | Old default | New default | Description | Affected components | | --- | --- | --- | --- | --- | | metadata_max_age | 5m | 1m | The default changed from 5m to 1m to provide more responsive topic and partition discovery. This means metadata refreshes occur more frequently by default. If you experience increased broker load or want to maintain the previous behavior, explicitly set this field to 5m. | Config:redpandaCaches:redpandaInputs:kafka_franzlegacy_redpanda_migratorlegacy_redpanda_migrator_offsetsredpandaredpanda_migratorOutputs:kafka_franzlegacy_redpanda_migratorlegacy_redpanda_migrator_offsetsredpandaredpanda_migrator | ## [](#version-4-68-0)Version 4.68.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.68.0). ### [](#component-updates-9)Component updates This release adds the following new components: - Processors: - [`jira`](../../components/processors/jira/): Executes Jira API queries based on input messages and returns structured results. The processor handles pagination, retries, and field expansion automatically. This release adds support for the following new fields: - Inputs components - [`redpanda`](../../components/inputs/redpanda/): - [`unordered_processing`](../../components/inputs/redpanda/#unordered_processing) - Configures partition consumers to allow parallel and therefore unordered processing of messages of any given partition. This allows for better utilization of processing threads and asynchronous publishing at the output level. The maximum parallelization of each partition is determined by the checkpoint\_limit field. ### [](#deprecations-3)Deprecations The `kafka`, `kafka_franz`, and `redpanda_common` inputs and outputs are now deprecated. Their functionality has been consolidated into the unified `redpanda` input and output components. Migrate existing configurations to use the new `redpanda` components for continued support and access to the latest features. ## [](#version-4-67-5)Version 4.67.5 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.67.5). ### [](#component-updates-10)Component updates This release adds the following new components: - Inputs: - [`microsoft_sql_server_cdc`](../../components/inputs/microsoft_sql_server_cdc/): Streams change data from a Microsoft SQL Server database into Redpanda Connect using Change Data Capture (CDC). This release adds support for the following new fields: - Inputs components - [`redpanda_migrator`](../../components/inputs/redpanda_migrator/) - [`schema_registry`](../../components/inputs/redpanda_migrator/#schema_registry) - Configuration for schema registry integration. Enables migration of schema subjects, versions, and compatibility settings between clusters. - Outputs components - [`redpanda_migrator`](../../components/outputs/redpanda_migrator/) - [`schema_registry`](../../components/outputs/redpanda_migrator/#schema_registry) - Configuration for schema registry integration. Enables migration of schema subjects, versions, and compatibility settings between clusters. - [`consumer_groups`](../../components/outputs/redpanda_migrator/#consumer_groups) - Whether consumer group offset migration is enabled. When disabled, no consumer group operations are performed. - [`topic_replication_factor`](../../components/outputs/redpanda_migrator/#topic_replication_factor) - The replication factor for created topics. If not specified, inherits the replication factor from source topics. Useful when migrating to clusters with different sizes. - [`sync_topic_acls`](../../components/outputs/redpanda_migrator/#sync_topic_acls) - Whether to synchronize topic ACLs from source to destination cluster. ACLs are transformed safely: ALLOW WRITE permissions are excluded, and ALLOW ALL is downgraded to ALLOW READ to prevent conflicts. - [`serverless`](../../components/outputs/redpanda_migrator/#serverless) - Enable serverless mode for Redpanda Cloud serverless clusters. This restricts topic configurations and schema features to those supported by Redpanda Serverless environments. ### [](#deprecations-4)Deprecations This release deprecates the individual migrator components and the bundle wrapper in favor of the unified `redpanda_migrator` architecture. **Deprecated components:** - `redpanda_migrator` input and output (renamed to `legacy_redpanda_migrator`) - `redpanda_migrator_offsets` input and output (renamed to `legacy_redpanda_migrator_offsets`) - `redpanda_migrator_bundle` input and output (now deprecated; these are part of the legacy migration architecture and internally depend on the deprecated `legacy_redpanda_migrator` and `legacy_redpanda_migrator_offsets` components) These components have been renamed from their previous versions (without the `legacy_` prefix) to preserve backward compatibility. The new `redpanda_migrator` components introduced in this release contain breaking changes. Existing configurations using the previous `redpanda_migrator` components should use these `legacy_` prefixed versions to maintain current functionality. Users should migrate from deprecated components to the new unified `redpanda_migrator` input/output pair. For detailed migration instructions, see [Migrate to the Unified Redpanda Migrator](../../guides/migration/migrate-unified-redpanda-migrator/). ## [](#version-4-66-0)Version 4.66.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.66.0). ### [](#component-updates-11)Component updates This release adds the following new components: - Outputs: - [`cyborgdb`](../../components/outputs/cyborgdb/): This output allows you to write vectors to a CyborgDB encrypted index. CyborgDB provides end-to-end encrypted vector storage with automatic dimension detection and index optimization. All vector data is encrypted client-side before being sent to the server, ensuring complete data privacy. The encryption key never leaves your infrastructure. - Bloblang-functions: - [`with_schema_registry_header`](../../guides/bloblang/functions/#with_schema_registry_header): Prepends a 5-byte Schema Registry header to a message. The header consists of a magic byte (0x00) followed by a 4-byte big-endian schema ID. This release adds support for the following new fields: - Inputs components - [`cassandra`](../../components/inputs/cassandra/) - [`reconnect_interval`](../../components/inputs/cassandra/#reconnect_interval) - Attempts to reconnect known DOWN nodes in every ReconnectInterval. - [`exponential_reconnection`](../../components/inputs/cassandra/#exponential_reconnection) - Optional exponential reconnection policy, this replaces the default constant policy of the driver. - Outputs components - [`cassandra`](../../components/outputs/cassandra/) - [`reconnect_interval`](../../components/outputs/cassandra/#reconnect_interval) - Attempts to reconnect known DOWN nodes in every ReconnectInterval. - [`exponential_reconnection`](../../components/outputs/cassandra/#exponential_reconnection) - Optional exponential reconnection policy, this replaces the default constant policy of the driver. ## [](#version-4-65-0)Version 4.65.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.65.0). ### [](#component-updates-12)Component updates This release adds the following new components: - Inputs: - [`tigerbeetle_cdc`](../../components/inputs/tigerbeetle_cdc/): Streams change data from a TigerBeetle database into Redpanda Connect. > 📝 **NOTE** > > The `tigerbeetle_cdc` input requires a CGO-enabled build of Redpanda Connect. > > For instructions, see: > > - [Download a CGO-enabled binary](../../install/prebuilt-binary/) > > - [Build Redpanda Connect from source](../../install/build-from-source/) - Scanners: - [`json_array`](../../components/scanners/json_array/): Scans a JSON array and emits each element as a separate message. ### [](#model-context-protocol-mcp-server)Model Context Protocol (MCP) server You can now build and run an MCP server directly in Redpanda Connect. The MCP server lets you expose your Redpanda Connect configurations as AI-consumable HTTP endpoints, making them discoverable and callable by AI clients such as Claude Code and other compatible agents. To get started, see [MCP Server Quickstart](../../ai-agents/mcp-server/quickstart/). ## [](#version-4-64-0)Version 4.64.0 [See the full release notes](https://github.com/redpanda-data/connect/releases/tag/v4.64.0). ### [](#component-updates-13)Component updates This release adds support for the following new fields and improvements: - Schema registry decode processor: - [`default_schema_id`](../../components/processors/schema_registry_decode/#default_schema_id) field for specifying a fallback schema ID when decoding messages without an explicit schema reference. ## [](#version-4-62-0)Version 4.62.0 ### [](#component-updates-14)Component updates This release adds support for the following new fields and improvements: - Kafka, Redpanda, and Franz input components: - [`timely_nacks_maximum_wait`](../../components/inputs/kafka/#timely_nacks_maximum_wait) field for improved message delivery guarantees and SLA enforcement. - [`timely_nacks_maximum_wait`](../../components/inputs/kafka_franz/#timely_nacks_maximum_wait) field for improved message delivery guarantees and SLA enforcement. - [`timely_nacks_maximum_wait`](../../components/inputs/redpanda/#timely_nacks_maximum_wait) field for improved message delivery guarantees and SLA enforcement. - Redpanda Migrator and Redpanda Common input components: - [`timely_nacks_maximum_wait`](../../components/inputs/redpanda_migrator/#timely_nacks_maximum_wait) field for consistent negative acknowledgement handling. - [`timely_nacks_maximum_wait`](../../components/inputs/redpanda_common/#timely_nacks_maximum_wait) field for consistent negative acknowledgement handling. - Schema Registry output component: - [`subject_compatibility_level`](../../components/outputs/schema_registry/#subject_compatibility_level) field for explicit subject compatibility configuration. - Parquet Encode processor: - [`schema_metadata`](../../components/processors/parquet_encode/#schema_metadata) field for dynamic schema assignment during encoding. - Socket output component: - [`Full TLS configuration support`](../../components/outputs/socket/#tls), including all standard and mTLS fields, now deduplicated and centrally documented. - Bloblang: - [`infer_schema`](../../guides/bloblang/methods/#infer_schema) method attempts to infer the schema of a given value. The resulting schema can then be used as an input to schema conversion and enforcement methods. ## [](#version-4-61-0)Version 4.61.0 ### [](#component-updates-15)Component updates This release adds support for the following new fields: - Cassandra input and output components - Input: [`host_selection_policy`](../../components/inputs/cassandra/#host_selection_policy) - Output: [`host_selection_policy`](../../components/outputs/cassandra/#host_selection_policy) - Schema Registry output component - [`normalize`](../../components/outputs/schema_registry/#normalize) - [`remove_metadata`](../../components/outputs/schema_registry/#remove_metadata) - [`remove_rule_set`](../../components/outputs/schema_registry/#remove_rule_set) ## [](#version-4-60-0)Version 4.60.0 ### [](#component-updates-16)Component updates The 4.60.0 release adds support for the following fields: - NATS components - `tls_handshake_first` - Enhanced TLS configuration option for NATS connections that require the initial TLS handshake to be performed before sending the NATS INFO protocol message. This is required when connecting to some NATS servers that expect TLS to be established immediately after connection, before any protocol negotiation. This field is available in the following NATS components: - [`nats` input](../../components/inputs/nats/#tls_handshake_first) - [`nats_jetstream` input](../../components/inputs/nats_jetstream/#tls_handshake_first) - [`nats_kv` input](../../components/inputs/nats_kv/#tls_handshake_first) - [`nats_stream` input](../../components/inputs/nats_stream/#tls_handshake_first) - [`nats` output](../../components/outputs/nats/#tls_handshake_first) - [`nats_jetstream` output](../../components/outputs/nats_jetstream/#tls_handshake_first) - [`nats_kv` output](../../components/outputs/nats_kv/#tls_handshake_first) - [`nats_stream` output](../../components/outputs/nats_stream/#tls_handshake_first) - [`nats_kv` processor](../../components/processors/nats_kv/#tls_handshake_first) - [`nats_request_reply` processor](../../components/processors/nats_request_reply/#tls_handshake_first) - [`nats_kv` cache](../../components/caches/nats_kv/#tls_handshake_first) ## [](#version-4-59-0)Version 4.59.0 ### [](#redpanda-connect-in-redpanda-cloud-ga)Redpanda Connect in Redpanda Cloud: GA Redpanda Connect is now generally available (GA) in all Redpanda Cloud clusters: BYOC (including BYOVPC/BYOVNet), Dedicated, and Serverless. Redpanda Connect has been GA in Redpanda Self-Managed clusters since version 4.29.0. ### [](#component-updates-17)Component updates This release adds support for the following fields: - SFTP input components - [`connection_timeout`](../../components/inputs/sftp/#connection_timeout) - [`credentials.host_public_key`](../../components/inputs/sftp/#credentials-host_public_key) - [`credentials.host_public_key_file`](../../components/inputs/sftp/#credentials-host_public_key_file) - [`max_sftp_sessions`](../../components/inputs/sftp/#max_sftp_sessions) - SFTP output components - [`connection_timeout`](../../components/outputs/sftp/#connection_timeout) - [`credentials.host_public_key`](../../components/outputs/sftp/#credentials-host_public_key) - [`credentials.host_public_key_file`](../../components/outputs/sftp/#credentials-host_public_key_file) - GCP Pub/Sub output components - [`validate_topic`](../../components/outputs/gcp_pubsub/#validate_topic) - Redpanda cache components - [`allow_auto_topic_creation`](../../components/caches/redpanda/#allow_auto_topic_creation) ## [](#version-4-58-2)Version 4.58.2 This release introduces an experimental Slack output and expands support for automatic topic creation across Redpanda-related components. ### [](#new-components)New components - [`slack_reaction`](../../components/outputs/slack_reaction/): Adds or removes emoji reactions from Slack messages by channel ID and timestamp. ### [](#component-updates-18)Component updates This release adds support for the `allow_auto_topic_creation` field in the following components: - [`redpanda`](../../components/redpanda/about/) configuration block. - [`kafka_franz`](../../components/outputs/kafka_franz/) output. - [`redpanda`](../../components/outputs/redpanda/) output. - [`redpanda_migrator`](../../components/outputs/redpanda_migrator/) output. This field allows components to create topics automatically when publishing data, if the underlying Kafka/Redpanda broker is configured to allow it. ## [](#version-4-57-0)Version 4.57.0 This release adds support for the following fields: - `protobuf.use_proto_names`, `protobuf.use_enum_numbers`, `protobuf.emit_unpopulated`, and `protobuf.emit_default_values` in the [`schema_registry_decode` processor](../../components/processors/schema_registry_decode/). ## [](#version-4-56-0)Version 4.56.0 This release introduces a dynamic [plugins](../../plugins/about/) framework. It enables you to create and load custom plugins at runtime using any programming language that supports gRPC, with initial SDKs for Go and Python. ### [](#new-components-2)New components - [`gcp_spanner_cdc`](../../components/inputs/gcp_spanner_cdc/): Streams change data from a Google Cloud Spanner instance. ### [](#component-updates-19)Component updates This release adds support for the following fields: - `scope` in the [`couchbase`](../../components/caches/couchbase/) cache. - `poll_interval` and `max_yield_batch_bytes` in the `redpanda_migrator_offsets` input. - `consumer_group_offsets_poll_interval` in the `redpanda_migrator_bundle` input. - `input_bundle_label` in the `redpanda_migrator_bundle` output. - `object_canned_acl` in the [`aws_s3`](../../components/outputs/aws_s3/) output. - `history`, `max_tool_calls`, and `tools` in the [`gcp_vertex_ai_chat`](../../components/processors/gcp_vertex_ai_chat/) processor. ## [](#version-4-55-1)Version 4.55.1 This release adds support for the following fields: - `is_serverless` in the [`redpanda_migrator`](../../components/outputs/redpanda_migrator/) output. ## [](#version-4-55-0)Version 4.55.0 This release introduces a new beta Redpanda cache component and new fields across several inputs and outputs. ### [](#new-components-3)New components - [`redpanda`](../../components/caches/redpanda/): A new cache that stores key/value pairs in a compacted Redpanda topic. ### [](#component-updates-20)Component updates This release adds support for the following fields: - `steal_grace_period` in the [`aws_kinesis`](../../components/inputs/aws_kinesis/) input. - `max_yield_batch_bytes` in: - [`redpanda`](../../components/inputs/redpanda/) - [`redpanda_common`](../../components/inputs/redpanda_common/) - [`redpanda_migrator`](../../components/inputs/redpanda_migrator/) input - [`redpanda_migrator`](../../components/outputs/redpanda_migrator/) output - `extras` in the [`sentry_capture`](../../components/processors/sentry_capture/) processor. ### [](#removed-components-2)Removed components The deprecated `elasticsearch` output has been removed. Use the [`elasticsearch_v8`](../../components/outputs/elasticsearch_v8/) output instead. ### [](#new-bloblang-features)New Bloblang features - [`bitwise_and`](../../guides/bloblang/methods/#bitwise_and), [`bitwise_or`](../../guides/bloblang/methods/#bitwise_or), and [`bitwise_xor`](../../guides/bloblang/methods/#bitwise_xor) methods added for performing bitwise operations on integers. ## [](#version-4-54-0-to-4-54-1)Version 4.54.0 to 4.54.1 These releases introduce the [`qdrant` processor](../../components/processors/qdrant/), which allows you to query items within a Qdrant collection and filter the returned results. ### [](#component-updates-21)Component updates These releases add: - The `topic_lag_refresh_period` field to the [`kafka_franz`](../../components/inputs/kafka_franz/) and [`ockam_kafka`](../../components/inputs/ockam_kafka/) inputs. - The `cache_duration` field to the [`schema_registry_decode` processor](../../components/processors/schema_registry_decode/). - The `client_auth` field to the [`socket_server` input](../../components/inputs/socket_server/). You can also now use Bloblang to generate a [name-based UUID (version 5)](../../guides/bloblang/methods/#uuid_v5) for a given string. ## [](#version-4-53-0)Version 4.53.0 This release introduces three new Google Drive processors: - [`google_drive_search` processor](../../components/processors/google_drive_search/): Searches Google Drive for files that match a specified query and emits the results as a batch of messages. - [`google_drive_download` processor](../../components/processors/google_drive_download/): Downloads files from Google Drive that contain matching file IDs. - [`google_drive_list_labels` processor](../../components/processors/google_drive_list_labels/): Lists labels for files on a Google Drive. The release also adds the [`cohere_rerank` processor](../../components/processors/cohere_rerank/), which sends document strings to the Cohere API, and returns them ranked by their relevance to a specified query. ### [](#component-updates-22)Component updates This release adds: - The `transaction_isolation_level` field to the following inputs: [`kafka_franz`](../../components/inputs/kafka_franz/), [`ockam_kafka`](../../components/inputs/ockam_kafka/), [`redpanda`](../../components/inputs/redpanda/), [`redpanda_common`](../../components/inputs/redpanda_common/), [`redpanda_migrator`](../../components/inputs/redpanda_migrator/). - The `request_timeout_overhead` and `conn_idle_timeout` fields to the following components: [`kafka_franz` input](../../components/inputs/kafka_franz/), [`redpanda` input](../../components/inputs/redpanda/), [`redpanda_migrator` input](../../components/inputs/redpanda_migrator/), [`kafka_franz` output](../../components/outputs/kafka_franz/), [`redpanda` output](../../components/outputs/redpanda/), [`redpanda_migrator` output](../../components/outputs/redpanda_migrator/), `redpanda_migrator_offsets` output, [`redpanda` configuration service](../../components/redpanda/about/). - The `start_offset` field to the following components: [`kafka_franz` input](../../components/inputs/kafka_franz/), [`ockam_kafka` input](../../components/inputs/ockam_kafka/), [`redpanda` input](../../components/inputs/redpanda/), [`redpanda_common` input](../../components/inputs/redpanda_common/), [`redpanda_migrator` input](../../components/inputs/redpanda_migrator/). - The `topic_prefix` field to the [`redpanda_migrator` output](../../components/outputs/redpanda_migrator/), and the `offset_topic_prefix` field to the `redpanda_migrator_offsets` output, to more easily identify migrated topics. - The `use_enum_numbers` field to the [`protobuf` processor](../../components/processors/protobuf/). - The `dimensions` field to the [`cohere_embeddings` processor](../../components/processors/cohere_embeddings/). - The `region`, `endpoint`, and `credentials` fields to the `dynamodb` section of the [`aws_kinesis` input](../../components/inputs/aws_kinesis/). - A `tools` section to the [`cohere_chat` processor](../../components/processors/cohere_chat/) to enhance response generation with agentic-like behavior. ### [](#deprecations-5)Deprecations This release deprecates the `start_from_oldest` field from the following components: [`kafka_franz` input](../../components/inputs/kafka_franz/), [`ockam_kafka` input](../../components/inputs/ockam_kafka/), [`redpanda` input](../../components/inputs/redpanda/), [`redpanda_common` input](../../components/inputs/redpanda_common/), [`redpanda_migrator` input](../../components/inputs/redpanda_migrator/). Now, you must use the `start_offset` field instead, which defaults to `earliest`. Redpanda recommends upgrading to benefit from this update. ## [](#version-4-52-0)Version 4.52.0 This release introduces three new Slack components: - [`slack_users` input](../../components/inputs/slack_users/): Reads the full profiles of users in a Slack organization. - [`slack_thread` processor](../../components/processors/slack_thread/): Reads all messages in a Slack thread. - [`slack_post` output](../../components/outputs/slack_post/): Posts messages to Slack channel. ## [](#version-4-51-0)Version 4.51.0 Highlights of this update include the introduction of three new components that you could use as part of an AI toolchain: - [`git` input](../../components/inputs/git/) - [`text_chunker` processor](../../components/processors/text_chunker/) - [`slack` input](../../components/inputs/slack/) You can also now [install a FIPS-compliant version of Redpanda Connect](../quickstarts/rpk/#fips-compliance) (for RHEL/Debian Ubuntu). ### [](#component-updates-23)Component updates This release adds: - The `private_key` field to the [`sftp` input](../../components/inputs/sftp/) and [output](../../components/outputs/sftp/) - The `history` field to the [`ollama_chat`](../../components/processors/ollama_chat/) and [`openai_chat_completion`](../../components/processors/openai_chat_completion/) processors, which allows you to include historical messages in a chat request. - The `handle_logical_types` fields to the [`parquet_decode` processor](../../components/processors/parquet_decode/). - The `aggregate` operation to the [`mongodb` processor](../../components/processors/mongodb/), which provides support for aggregation pipelines. - The `headers` option to the `type` field in the [`amqp_0_9` output](../../components/outputs/amqp_0_9/). ## [](#version-4-50-0)Version 4.50.0 This release introduces agentic behavior to the [`openai_chat_completion` processor](../../components/processors/openai_chat_completion/). You can now specify external tools to enrich the responses of OpenAI large language models. ### [](#component-updates-24)Component updates This release: - Adds a set of `tools` fields to the [`openai_chat_completion` processor](../../components/processors/openai_chat_completion/). - Extends support for FLOAT values in the [`snowflake_streaming` output](../../components/outputs/snowflake_streaming/) to include special values, such as NaN (Not a Number), -inf (negative infinity), and inf (positive infinity). Redpanda Connect also now supports the [`unicode_segments` Bloblang method](../../guides/bloblang/methods/#unicode_segments), which splits a string into segments using [Unicode text segmentation rules](https://hexdocs.pm/unicode_string/readme.html#segmentation). ## [](#version-4-49-1)Version 4.49.1 This release introduces further updates to the [`postgres_cdc` input](../../components/inputs/postgres_cdc/) and [`snowflake_streaming` output](../../components/outputs/snowflake_streaming/). ### [](#component-updates-25)Component updates This release adds: - Two new metrics to the [`snowflake_streaming` output](../../components/outputs/snowflake_streaming/): `snowflake_register_latency_ns` and `snowflake_commit_latency_ns`. - An `ignore_nulls` option for schema evolution in the [`snowflake_streaming` output](../../components/outputs/snowflake_streaming/). - The `translate_ids` field to the [`schema_registry` output](../../components/outputs/schema_registry/). - The `translate_schema_ids` field to the `redpanda_migrator_bundle` output. Also, the `translate_schema_ids` field in the [`redpanda_migrator` output](../../components/outputs/redpanda_migrator/) now defaults to `false`. #### [](#deprecations-6)Deprecations This release deprecates the `snapshot_memory_factor` field from the [`postgres_cdc` input](../../components/inputs/postgres_cdc/). Now, you must set a batch size, which defaults to `1000` messages. Redpanda recommends upgrading to benefit from this update. ## [](#version-4-48-1)Version 4.48.1 This release introduces a [change data capture (CDC) input for MongoDB](../../components/inputs/mongodb_cdc/) on Cloud and Self-Managed Redpanda Connect, along with improvements to the [`postgres_cdc` input](../../components/inputs/postgres_cdc/) and [`snowflake_streaming` output](../../components/outputs/snowflake_streaming/). ### [](#component-updates-26)Component updates This release adds: - The `heartbeat_interval` field to the [`postgres_cdc` input](../../components/inputs/postgres_cdc/), which helps manage the size of the Write-Ahead Log when you subscribe to data changes from tables with low activity. - The `commit_timeout` field to the [`snowflake_streaming` output](../../components/outputs/snowflake_streaming/). - The `url` field to the [`snowflake_streaming` output](../../components/outputs/snowflake_streaming/), which allows you to override the default connection URL. - The `is_high_watermark` field to the `redpanda_migrator_offsets` output. - The `kafka_is_high_watermark` metadata field to the `redpanda_migrator_offsets` input. Other changes: - You can now [load an Enterprise Edition license](../licensing/) using the `REDPANDA_LICENSE` environment variable. - The [`postgres_cdc` input](../../components/inputs/postgres_cdc/) no longer adds the prefix `_rs` to the replication slots it creates. - The `clickhouse` driver is now supported in Redpanda Cloud for all components with the prefix `sql_`. - The [`redpanda migrator` output](../../components/outputs/redpanda_migrator/) now attempts to transfer access control lists for topics that already exist on the destination broker. ## [](#version-4-47-0-to-4-47-1)Version 4.47.0 to 4.47.1 This release introduces: - The [`elasticsearch_v8` output](../../components/outputs/elasticsearch_v8/): A certified, up-to-date alternative to the existing `elasticsearch` output. - The [`crash` processor](../../components/processors/crash/): A tool for detecting unhandled pipeline errors during development and testing. ### [](#component-updates-27)Component updates This release adds: - The `arguments` field to the [`amqp_0_9` input](../../components/inputs/amqp_0_9/) and [output](../../components/outputs/amqp_0_9/). - The `mapping` field (for Avro messages) to the [`schema_registry_decode` processor](../../components/processors/schema_registry_decode/). - The `disable_http2` field to the [`http_client` input](../../components/inputs/http_client/), [output](../../components/outputs/http_client/), and [`http` processor](../../components/processors/http/). - The `retry_on_conflict` field to the [`elasticsearch` output](../../components/outputs/elasticsearch_v8/). Redpanda Connect also now supports the [`uuid_v7` Bloblang function](../../guides/bloblang/functions/#uuid_v7), which is useful for scenarios when the exact timing of events is important, such as during data migration or replication. ## [](#version-4-46-0)Version 4.46.0 This release introduces a [change data capture (CDC) input for MySQL](../../components/inputs/mysql_cdc/), and enhancements to the [Bloblang playground](../../guides/bloblang/playground/), making it easier to test and debug your pipeline configurations. The enhancements include: - A **Run** button in Bloblang code snippets, which opens each snippet in the playground with a pre-filled input and mapping ready for testing. - A **Get help from AI** button, which appears in the playground when an error occurs. ### [](#components-for-cloud-and-self-managed-redpanda-connect-beta)Components for Cloud and Self-Managed Redpanda Connect (beta) - [`mysql_cdc` input](../../components/inputs/mysql_cdc/) ### [](#component-updates-28)Component updates This release adds: - The `instance_id` field to the [`kafka`](../../components/inputs/kafka/), [`kafka_franz`](../../components/inputs/kafka_franz/), [`ockam_kafka`](../../components/inputs/ockam_kafka/), [`redpanda`](../../components/inputs/redpanda/), [`redpanda_common`](../../components/inputs/redpanda_common/), and [`redpanda_migrator`](../../components/inputs/redpanda_migrator/) inputs. - The `rebalance_timeout`, `session_timeout`, and `heartbeat_interval` fields to the [`kafka_franz`](../../components/inputs/kafka_franz/), [`ockam_kafka`](../../components/inputs/ockam_kafka/), [`redpanda`](../../components/inputs/redpanda/), [`redpanda_common`](../../components/inputs/redpanda_common/), and [`redpanda_migrator`](../../components/inputs/redpanda_migrator/) inputs. - The `preserve_logical_types` and `raw_unions` fields (for Avro messages), and a `schema_id` metadata field to the [`schema_registry_decode`](../../components/processors/schema_registry_decode/) processor. - The `unchanged_toast_value` field to the [`postgres_cdc` input](../../components/inputs/postgres_cdc/). - A `processors` field to the `schema_evolution` object in the [`snowflake_streaming` output](../../components/outputs/snowflake_streaming/). - New default values to the `max_message_bytes` and `broker_write_max_bytes` fields in the [`kafka_franz`](../../components/outputs/kafka_franz/), [`ockam_kafka`](../../components/outputs/ockam_kafka/), [`redpanda`](../../components/outputs/redpanda/), [`redpanda_migrator`](../../components/outputs/redpanda_migrator/), and `redpanda_migrator_offsets` outputs, using IEC instead of SI units. Other changes: - The [`postgres_cdc` input](../../components/inputs/postgres_cdc/) no longer supports PostgreSQL 10 or 11. #### [](#deprecations-7)Deprecations All changes are backward-compatible. No action is required if you are using an earlier version of Redpanda Connect. | Deprecated fields | Affected components | | --- | --- | | new_column_type_mapping | snowflake_streaming output | ## [](#version-4-45-0)Version 4.45.0 Highlights of this update include the beta release of two components for Self-Managed and Cloud Redpanda Connect, and [a new cookbook](../../cookbooks/snowflake_ingestion/) that explains how to ingest data from a Redpanda topic into Snowflake using Snowpipe Streaming. You can also now share links to code snippets in the [Bloblang playground](../../guides/bloblang/playground/) using the **Share link** button. ### [](#components-for-cloud-and-self-managed-redpanda-connect-beta-2)Components for Cloud and Self-Managed Redpanda Connect (beta) - `ollama_moderation` processor ([Self-Managed](../../components/processors/ollama_moderation/), [Cloud](../../../redpanda-cloud/develop/connect/components/processors/ollama_moderation/)) - `redpanda_migrator_offsets` input (`Self-Managed`, `Cloud`) ### [](#component-updates-29)Component updates This release adds: - The `max_outstanding_messages` and `message_timeout` fields to the [`aws_sqs` input](../../components/inputs/aws_sqs/). - The `content_type` field to the [`amqp_1` output](../../components/outputs/amqp_1/). - The `offset_token` field to the [`snowflake_streaming` output](../../components/outputs/snowflake_streaming/). - The `save_prompt_metadata`, `max_tool_calls` fields and `tools` object to the [`ollama_chat` processor](../../components/processors/ollama_chat/). - The `offset_topic`, `offset_group`, `offset_partition`, `offset_commit_timestamp` and `offset_metadata` fields to the `redpanda_migrator_offsets` output. - The `topic_lag_refresh_period` field to the [`redpanda`](../../components/inputs/redpanda/) and [`redpanda_common`](../../components/inputs/redpanda_common/) inputs. - The `redpanda_lag` metric to the [`redpanda`](../../components/inputs/redpanda/) and [`redpanda_common`](../../components/inputs/redpanda_common/) inputs. - The `fetch_max_wait` field to the [`kafka_franz`](../../components/inputs/kafka_franz/), [`ockam_kafka`](../../components/inputs/ockam_kafka/), [`redpanda`](../../components/inputs/redpanda/), [`redpanda_common`](../../components/inputs/redpanda_common/), and [`redpanda_migrator`](../../components/inputs/redpanda_migrator/) inputs. - The `kafka_lag` metadata field to the [`redpanda`](../../components/inputs/redpanda/) and [`redpanda_common`](../../components/inputs/redpanda_common/) inputs. - The `avro_schema` and `avro_schema_fingerprint` metadata fields to the [`avro` scanner](../../components/scanners/avro/). - The `label` field to [template test definitions](../../configuration/templating/). - The `bloblang` scalar type to [template fields](../../configuration/templating/). Redpanda Connect now supports: - Interpolation functions for channel and table names in the [`snowflake_streaming` output](../../components/outputs/snowflake_streaming/). - Calls to external tools when using the [`ollama_chat` processor](../../components/processors/ollama_chat/). - The execution of multiple database queries as transactions when using the [`sql_raw` input](../../components/inputs/sql_raw/), [output](../../components/outputs/sql_raw/) and [processor](../../components/processors/sql_raw/). - Using the [`label` metadata field](../../configuration/templating/) within a template mapping to retrieve the template component `label` set in the corresponding configuration. - The following Bloblang functions for error handling: [`error_source_name`](../../guides/bloblang/functions/#error_source_name), [`error_source_label`](../../guides/bloblang/functions/#error_source_label), and [`error_source_path`](../../guides/bloblang/functions/#error_source_path). #### [](#deprecations-8)Deprecations All changes are backward-compatible. No action is required if you are using an earlier version of Redpanda Connect. | Deprecated fields | Affected components | | --- | --- | | batching | redpanda_migrator input, redpanda_migrator output | | kafka_key, max_in_flight | redpanda_migrator_offsets output | | batch_size, multi_header, replication_factor, replication_factor_override, output_resource | redpanda_migrator input | ## [](#version-4-44-0)Version 4.44.0 This release extends the functionality of the [`openai_chat_completion` processor](../../components/processors/openai_chat_completion/) to include the following fields: `max_tokens`, `temperature`, `user`, `top_p`, `frequency_penalty`, `presence_penalty`, `seed`, and `stop`. ## [](#version-4-43-1)Version 4.43.1 This release introduces support for [trial licenses](../licensing/) for Self-Managed Redpanda Connect. ## [](#version-4-43-0)Version 4.43.0 This release renames the `pg_stream` input to [`postgres_cdc`](../../components/inputs/postgres_cdc/). The `postgres_cdc` input also no longer emits `mode` metadata, but sets the `operation` metadata field to `read` when a snapshot of a database is processed. ## [](#version-4-42-0)Version 4.42.0 This release introduces a [Bloblang playground](../../guides/bloblang/playground/) where you can try out your Bloblang syntax and get immediate feedback. It also updates Redpanda Connect licensing checks for Redpanda Connect. Starting from version 4.42.0, you must [add your Enterprise Edition license](../licensing/) to Redpanda Connect before using any Enterprise connectors. ### [](#component-updates-30)Component updates Redpanda Connect now supports: - A Google Cloud `spanner` driver for all SQL plugins. - The following complex data types for the [`pg_stream` input](../../components/inputs/postgres_cdc/): JSONB, TEXT\[\], INET, TSVECTOR, TSRANGE, POINT, INTEGER\[\]. - Parquet files for the [`bigquery` output](../../components/inputs/gcp_bigquery_select/). This release also adds an `exists` operator to the [`cache` processor](../../components/processors/cache/). ## [](#version-4-41-0)Version 4.41.0 This release adds the `max_records_per_request` field to the [`aws_sqs` output](../../components/outputs/aws_sqs/). ### [](#packaging-update)Packaging update Starting from version 4.41.0, the `-cgo` suffixed Docker image is no longer available due to low demand and the unacceptable cadence with which the image base (Debian) receives security updates. To create your own CGO builds, run the following command: ```bash CGO_ENABLED=1 make TAGS=x_benthos_extra redpanda-connect. ``` ## [](#version-4-40-0)Version 4.40.0 This update includes a beta release of: - The [`pg_stream` input](../../components/inputs/postgres_cdc/), which captures data changes made to a PostgreSQL database in real time and streams them to Redpanda Connect. - The [`benchmark` processor](../../components/processors/benchmark/), which provides message throughput statistics for a data pipeline. It also introduces a new [Bloblang method for timestamps](../../guides/bloblang/methods/#timestamp). ### [](#components-for-cloud-and-self-managed-redpanda-connect-beta-3)Components for Cloud and Self-Managed Redpanda Connect (beta) - [`pg_stream` input](../../components/inputs/postgres_cdc/) - [`benchmark` processor](../../components/processors/benchmark/) ### [](#component-updates-31)Component updates This release adds: - The `metadata_max_age` field to the `redpanda_migrator_offsets` output. - The `timestamp_ms` field to the following outputs: [`ockam_kafka`](../../components/outputs/ockam_kafka/), [`kafka`](../../components/outputs/kafka/), [`kafka_franz`](../../components/outputs/kafka_franz/), [`redpanda`](../../components/outputs/redpanda/), [`redpanda_common`](../../components/outputs/redpanda_common/), [`redpanda_migrator`](../../components/outputs/redpanda_migrator/). - The `kafka_timestamp_ms` metadata field, which uses a Unix timestamp with millisecond precision, to the following inputs: [`kafka`](../../components/inputs/kafka/), [`kafka_franz`](../../components/inputs/kafka_franz/), [`redpanda`](../../components/inputs/redpanda/), [`redpanda_common`](../../components/inputs/redpanda_common/), [`redpanda_migrator`](../../components/inputs/redpanda_migrator/). - The `translate_schema_ids` and `schema_registry_output_resource` to the [`redpanda_migrator` output](../../components/outputs/redpanda_migrator/). - The `backfill_dependencies` and `input_resource` fields to the [`schema_registry` output](../../components/outputs/schema_registry/). - Options to optimize the build of the data output from the [`snowflake_streaming` output](../../components/outputs/snowflake_streaming/). Other changes: - The following components now use the [Franz Kafka Schema Registry client library](https://github.com/twmb/franz-go/tree/master/pkg/sr): [`schema_registry` input](../../components/inputs/schema_registry/) and [output](../../components/outputs/schema_registry/), and the [`schema_registry_encode`](../../components/processors/schema_registry_encode/) and [`schema_registry_decode`](../../components/processors/schema_registry_decode/) processors. - The [`snowflake_streaming` output](../../components/outputs/snowflake_streaming/) can now automatically create tables in Snowflake, when the `schema_evolution.enabled` field is set to `true`. #### [](#deprecations-9)Deprecations **Deprecated field**: `kafka_timestamp` metadata field **Affected components**: - [`redpanda` input](../../components/inputs/redpanda/) - [`redpanda_common` input](../../components/inputs/redpanda_common/) **Replaced by**: `kafka_timestamp_ms` metadata field **Deprecated field**: `timestamp` **Affected components**: - [`ockam_kafka` output](../../components/outputs/ockam_kafka/) - [`kafka` output](../../components/outputs/kafka/) - [`kafka_franz` output](../../components/outputs/kafka_franz/) - [`redpanda` output](../../components/outputs/redpanda/) - [`redpanda_common` output](../../components/outputs/redpanda_common/) - [`redpanda_migrator` output](../../components/outputs/redpanda_migrator/) **Replaced by**: `timestamp_ms` field **Action required**: Modify all systems that process these fields to ensure compatibility with the new field names. ## [](#version-4-39-0)Version 4.39.0 Highlights of this update include the beta release of three new components for Self-Managed and Cloud, including a new `snowflake_streaming` output. This release also includes additions to the `redpanda` family of components and introduces new features for Self-Managed. ### [](#components-for-cloud-and-self-managed-beta)Components for Cloud and Self-Managed (beta) - [`snowflake_streaming` output](../../components/outputs/snowflake_streaming/) - [`timeplus` input](../../components/inputs/timeplus/) - [`spicedb_watch` input](../../components/inputs/spicedb_watch/) ### [](#additional-redpanda-components-beta)Additional `redpanda` components (beta) - `redpanda` ([input](../../components/inputs/redpanda/) and [output](../../components/outputs/redpanda/)): Consume message data from or send message data to Kafka brokers. - `redpanda_common` ([input](../../components/inputs/redpanda_common/) and [output](../../components/outputs/redpanda_common/)): Consume message data from or send message data to a Redpanda Kafka broker, using credentials from a common [`redpanda` configuration block](../../components/redpanda/about/) (Self-Managed only). ### [](#features-in-self-managed)Features in Self-Managed You can now: - Create an [allow list or deny list](../../configuration/allow_and_deny_lists/) for a Redpanda Connect instance - Use the [`rpk connect` CLI flag `--secrets`](../../configuration/secrets/#look-up-secrets-on-a-remote-system-at-runtime) to look up secrets on a remote system - Use the [`rpk connect` CLI flag `--disable-telemetry`](../../configuration/telemetry/#disable-the-telemetry-service) to block the telemetry service ### [](#component-updates-32)Component updates This release adds: - The `follow_redirects` field to the [`http` processor](../../components/processors/http/). - The following fields to the [`redpanda` configuration service](../../components/redpanda/about/): - `metadata_max_age` - `partitioner` - `idempotent_write` - `compression` - `broker_write_max_bytes` - The `metadata_max_age` field to the [`kafka_franz` output](../../components/outputs/kafka_franz/). - The following fields to the [`kafka_franz`](../../components/inputs/kafka_franz/) and [`ockam_kafka`](../../components/inputs/ockam_kafka/) inputs: - `fetch_max_bytes` - `fetch_min_bytes` - `fetch_max_partition_bytes` - The `url` field of the [`aws_sqs` output](../../components/outputs/aws_sqs/) now supports interpolation functions. #### [](#deprecations-10)Deprecations All changes are backward-compatible. No action is required if you are using an earlier version of Redpanda Connect. | Deprecated fields | Affected components | | --- | --- | | auto_replay_nacks, sasl, metadata_max_age | ockam_kafka input | | client_id, rack_id, sasl | ockam_kafka output | | rack_id | kafka_franz output | ## [](#version-4-38-0)Version 4.38.0 This update includes the beta release of two new outputs for Cloud and Self-Managed, and the introduction of a [telemetry service](../../configuration/telemetry/), which collects and sends anonymized usage statistics to Redpanda. ### [](#new-tooling)New tooling A new Helm chart for deploying Redpanda Connect in Kubernetes is now available. This chart simplifies the deployment and configuration of Redpanda Connect, allowing you to manage and scale Redpanda Connect instances in Kubernetes environments. To get started with the Helm chart, see [Get Started with the Redpanda Connect Helm Chart](../quickstarts/helm-chart/). Or, see the release notes for the chart on [GitHub](https://github.com/redpanda-data/helm-charts/releases/tag/connect-3.0.0). ### [](#new-outputs-beta)New outputs (beta) - [`azure_data_lake_gen2`](../../components/outputs/azure_data_lake_gen2/) (certified) - [`timeplus`](../../components/outputs/timeplus/) (community) ### [](#component-updates-33)Component updates This release adds: - The `checksum_algorithm` to the [`aws_s3` output](../../components/outputs/aws_s3/). - The `private_key` field to the [`snowflake_put` output](../../components/outputs/snowflake_put/). - The `nkey` field to the following components: - `nats` ([input](../../components/inputs/nats/), [output](../../components/outputs/nats/)) - `nats_jetstream` ([input](../../components/inputs/nats_jetstream/), [output](../../components/outputs/nats_jetstream/)) - `nats_kv` ([input](../../components/inputs/nats_kv/), [output](../../components/outputs/nats_kv/), [processor](../../components/processors/nats_kv/)) - `nats_stream` ([input](../../components/inputs/nats_stream/), [output](../../components/outputs/nats_stream/)) - `nats_request_reply` ([processor](../../components/processors/nats_request_reply/)) Other changes: - The `collection` field of the [`mongodb` output](../../components/outputs/mongodb/) now supports interpolation functions. - All components that used the deprecated `count` function, with a default value for the `path` field, now use the `counter` function. > 📝 **NOTE** > > If you have multiple components that execute a mapping using the deprecated `count` function with the original default value, and these counters need to cascade, there may be a change in behavior. To avoid this, explicitly define a value for the `path` field. ## [](#version-4-37-0)Version 4.37.0 Redpanda Connect has new bundled [Redpanda Migrator components](#redpanda-migrator-components-beta), more [AI processors](#ai-processors-beta) for Self-Managed and Cloud platforms, and three new [outputs](#certified-and-community-outputs-beta). ### [](#redpanda-migrator-components-beta)Redpanda Migrator components (beta) You can now move workloads from any Apache Kafka system to Redpanda using the `redpanda_migrator_bundle` input and `output`. For full details, see the Redpanda Migrator Cookbook for [Self-Managed](../../cookbooks/redpanda_migrator/) and [Cloud](../../../redpanda-cloud/develop/connect/cookbooks/redpanda_migrator/). ### [](#ai-processors-beta)AI processors (beta) Some new additions to Redpanda Connect’s growing list of AI processors: - [`aws_bedrock_embeddings`](../../components/processors/aws_bedrock_embeddings/) - [`cohere_chat`](../../components/processors/cohere_chat/) - [`cohere_embeddings`](../../components/processors/cohere_embeddings/) - [`gcp_vertex_ai_embeddings`](../../components/processors/gcp_vertex_ai_embeddings/) ### [](#certified-and-community-outputs-beta)Certified and community outputs (beta) - [`couchbase`](../../components/outputs/couchbase/) (community) - [`cypher`](../../components/outputs/cypher/) (community) - [`questdb`](../../components/outputs/questdb/) (certified) ### [](#component-updates-34)Component updates This release adds: - The `metadata_max_age` field to [`kafka_franz`](../../components/inputs/kafka_franz/) and [`redpanda_migrator`](../../components/inputs/redpanda_migrator/) inputs. - The `fetch_in_order` field to the [`schema_registry` input](../../components/inputs/schema_registry/). - The `encrypted_fields` fields to the `ockam_kafka` [input](../../components/inputs/ockam_kafka/) and [output](../../components/outputs/ockam_kafka/). ## [](#version-4-36-0)Version 4.36.0 You can now use the `replication_factor_override` and `replication_factor` fields of the [`redpanda_migrator` input](../../components/inputs/redpanda_migrator/) and [output](../../components/outputs/redpanda_migrator/) to set a replication factor for migrated topics. ## [](#versions-4-35-1-to-4-35-3)Versions 4.35.1 to 4.35.3 Redpanda Connect is now fully integrated with the Redpanda Cloud platform. This release adds Azure and GCP components to the growing list of available components. For a full list of components supported in Redpanda Cloud, see the [Connector Catalog](../../../redpanda-cloud/develop/connect/components/about/). ## [](#version-4-35-0)Version 4.35.0 Redpanda Connect now includes beta versions of Redpanda Migrator components: - [`redpanda_migrator` input](../../components/inputs/redpanda_migrator/) and [output](../../components/outputs/redpanda_migrator/) - `redpanda_migrator_bundle` input and `output` - `redpanda_migrator_offsets` output For full details, see the Redpanda Migrator Cookbook for [Self-Managed](../../cookbooks/redpanda_migrator/) and [Cloud](../../../redpanda-cloud/develop/connect/cookbooks/redpanda_migrator/). ### [](#component-updates-35)Component updates This release adds: - OAuth, basic authentication, and JSON Web Token fields to the [`schema_registry` input](../../components/inputs/schema_registry/) and [output](../../components/outputs/schema_registry/). - The `job_project` field to the [`gcp_bigquery` output](../../components/outputs/gcp_bigquery/). - The `broker_write_max_bytes` field to the [`kafka_franz`](../../components/outputs/kafka_franz/) and [`ockam_kafka`](../../components/outputs/ockam_kafka/) outputs. ## [](#version-4-34-0)Version 4.34.0 Redpanda Connect now includes beta versions of two new AI processors for Cloud and Self-Managed platforms: - [`aws_bedrock_chat`](../../components/processors/aws_bedrock_chat/) - [`gcp_vertex_ai_chat`](../../components/processors/gcp_vertex_ai_chat/) ## [](#version-4-33-0)Version 4.33.0 Redpanda Connect now includes beta versions of four new components. ### [](#for-self-managed-and-cloud-platforms)For Self-Managed and Cloud platforms - [`schema_registry` input](../../components/inputs/schema_registry/) and [output](../../components/outputs/schema_registry/) - [`qdrant` output](../../components/outputs/qdrant/) ### [](#for-self-managed-only)For Self-Managed only - [`ockam_kafka` input](../../components/inputs/ockam_kafka/) and [output](../../components/outputs/ockam_kafka/) You can also use the new Bloblang [`vector`](../../guides/bloblang/methods/) method to create vector embeddings for SQL databases. ### [](#component-updates-36)Component updates This release adds: - The `credentials_json` field to all GCP components. - The `content_md5` field to the [`aws_s3` output](../../components/outputs/aws_s3/). - The `send_ack` field to the [`nats` input](../../components/inputs/nats/). ## [](#version-4-32-1)Version 4.32.1 Redpanda Connect has a new batch of beta AI processors for Self-Managed and Cloud platforms. ### [](#for-self-managed-and-cloud-platforms-2)For Self-Managed and Cloud platforms - [`openai_chat_completion`](../../components/processors/openai_chat_completion/) - [`openai_embeddings`](../../components/processors/openai_embeddings/) - [`openai_image_generation`](../../components/processors/openai_image_generation/) - [`openai_speech`](../../components/processors/openai_speech/) - [`openai_transcription`](../../components/processors/openai_transcription/) - [`openai_translation`](../../components/processors/openai_translation/) ### [](#for-self-managed-only-2)For Self-Managed only - [`ollama_chat`](../../components/processors/ollama_chat/) - [`ollama_embeddings`](../../components/processors/ollama_embeddings/) ## [](#version-4-31-0)Version 4.31.0 Redpanda Connect now includes beta versions of two new components: - [`pinecone` output](../../components/outputs/pinecone/) - Redpanda Connect’s first AI component - [`redpanda_data_transform` processor](../../components/processors/redpanda_data_transform/) ### [](#component-updates-37)Component updates This release adds: - Custom TLS fields for the [`splunk` input](../../components/inputs/splunk/) and [`splunk_hec` output](../../components/outputs/splunk_hec/). - The `timestamp` field to [`kafka`](../../components/outputs/kafka/) and [`kafka_franz`](../../components/outputs/kafka_franz/) outputs. - The `status_topic` field to the [`redpanda` component](../../components/redpanda/about/). ## [](#version-4-30-0-to-4-30-1)Version 4.30.0 to 4.30.1 Redpanda Connect now includes a beta version of the [`splunk` input](../../components/inputs/splunk/). ## [](#version-4-29-0)Version 4.29.0 Redpanda Connect now separates packages into an enterprise bundle and a free bundle, which contains community and selected certified connectors. To learn more about licensing and support levels, see [components:about.adoc#support-levels](../../components/about/#support-levels). ### [](#component-updates-38)Component updates This release adds: - The `auth.oauth2.scope` field to the [`pulsar` input](../../components/inputs/pulsar/) and [output](../../components/outputs/pulsar/). - The `subscription_initial_position` field to the [`pulsar` input](../../components/inputs/pulsar/). --- # Page 364: Guides **URL**: https://docs.redpanda.com/redpanda-connect/guides.md --- # Guides --- title: Guides latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: index page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: index.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/index.adoc page-git-created-date: "2024-06-03" page-git-modified-date: "2024-07-16" --- - [Monitoring](monitoring/) Learn how to monitor Redpanda Connect, including setting up metrics and alerts to ensure system health and performance. - [Message Delivery Semantics](delivery_semantics/) Learn about Redpanda Connect's transactional model and error handling - [Performance Tuning](performance_tuning/) Discover best practices for tuning Redpanda Connect performance, optimizing configurations, and achieving maximum throughput. - [Synchronous Responses](sync_responses/) Understand synchronous response handling in Redpanda Connect, ensuring reliable and efficient data processing. - [Cloud Credentials](cloud/) Explore how to deploy and manage Redpanda Connect in the cloud, including setup, configuration, and optimization tips. - [Serverless](serverless/about/) Learn about serverless architecture in Redpanda Connect, including benefits, use cases, and implementation details. - [Streams Mode](streams_mode/about/) Get an overview of streams mode in Redpanda Connect, detailing its features, use cases, and setup instructions. - [Migration](migration/) Migrate between different Redpanda Connect versions, including preparation steps and best practices. --- # Page 365: Bloblang **URL**: https://docs.redpanda.com/redpanda-connect/guides/bloblang/about.md --- # Bloblang --- title: Bloblang latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: bloblang/about page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: bloblang/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/bloblang/about.adoc description: Learn what Bloblang is and how to use the native mapping language. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-03-13" --- Bloblang, or blobl for short, is a language designed for mapping data of a wide variety of forms. It’s a safe, fast, and powerful way to perform document mapping within Redpanda Connect. It also has a [Go API for writing your own functions and methods](https://pkg.go.dev/github.com/redpanda-data/connect/v4/public/bloblang) as plugins. Bloblang is available as a [processor](../../../components/processors/mapping/) and it’s also possible to use blobl queries in [function interpolations](../../../configuration/interpolation/#bloblang-queries). You can also execute Bloblang mappings on the command-line with the `blobl` subcommand: ```shell cat data.jsonl | rpk connect blobl 'foo.(bar | baz).buz' ``` This document outlines the core features of the Bloblang language, but if you’re totally new to Bloblang then it’s worth following [the walkthrough first](../walkthrough/). ## [](#learn-bloblang)Learn Bloblang [learnbloblang.com](https://www.learnbloblang.com) is an interactive resource for learning Bloblang with hands-on exercises. ## [](#assignment)Assignment A Bloblang mapping expresses how to create a new document by extracting data from an existing input document. Assignments consist of a dot separated path segments on the left-hand side describing a field to be created within the new document, and a right-hand side query describing what the content of the new field should be. The keyword `root` on the left-hand side refers to the root of the new document, the keyword `this` on the right-hand side refers to the current context of the query, which is the read-only input document when querying from the root of a mapping: ```bloblang root.id = this.thing.id root.type = "yo" # Both `root` and `this` are optional, and will be inferred in their absence. content = thing.doc.message # In: {"thing":{"id":"wat1","doc":{"title":"wut","message":"hello world"}}} ``` Since the document being created starts off empty it is sometimes useful to begin a mapping by copying the entire contents of the input document, which can be expressed by assigning `this` to `root`. ```bloblang root = this root.foo = "added value" # In: {"id":"wat1","message":"hello world"} ``` If the new document `root` is never assigned to or otherwise mutated then the original document remains unchanged. ### [](#special-characters-in-paths)Special characters in paths Quotes can be used to describe sections of a field path that contain whitespace, dots or other special characters: ```bloblang # Use quotes around a path segment in order to include whitespace or dots within # the path root."foo.bar".baz = this."buz bev".fub # In: {"buz bev":{"fub":"hello world"}} ``` ### [](#non-structured-data)Non-structured data Bloblang is able to map data that is unstructured, whether it’s a log line or a binary blob, by referencing it with the [`content` function](../functions/#content), which returns the raw bytes of the input document: ```bloblang # Parse a base64 encoded JSON document root = content().decode("base64").parse_json() # In: eyJmb28iOiJiYXIifQ== ``` And your newly mapped document can also be unstructured, simply assign a value type to the `root` of your document: ```bloblang root = this.foo # In: {"foo":"hello world"} ``` And the resulting message payload will be the raw value you’ve assigned. ### [](#deleting)Deleting It’s possible to selectively delete fields from an object by assigning the function `deleted()` to the field path: ```bloblang root = this root.bar = deleted() # In: {"id":"wat1","message":"hello world","bar":"remove me"} ``` ### [](#variables)Variables Another type of assignment is a `let` statement, which creates a variable that can be referenced elsewhere within a mapping. Variables are discarded at the end of the mapping and are mostly useful for query reuse. Variables are referenced within queries with `$`: ```bloblang # Set a temporary variable let foo = "yo" root.new_doc.type = $foo ``` ### [](#metadata)Metadata Redpanda Connect messages contain metadata that is separate from the main payload, in Bloblang you can modify the metadata of the resulting message with the `meta` assignment keyword. Metadata values of the resulting message are referenced within queries with the `@` operator or the [`metadata()` function](../functions/#metadata): ```bloblang # Reference a metadata value root.new_doc.bar = @kafka_topic # Or `@.kafka_topic` or `metadata("kafka_topic")` # Delete all metadata meta = deleted() # Set metadata values meta bar = "hello world" meta baz = { "something": "structured" } # Get an object of key/values for all metadata root.meta_obj = @ # Or `metadata()` ``` ## [](#coalesce)Coalesce The pipe operator (`|`) used within brackets allows you to coalesce multiple candidates for a path segment. The first field that exists and has a non-null value will be selected: ```bloblang root.new_doc.type = this.thing.(article | comment | this).type # In: {"thing":{"article":{"type":"foo"}}} # In: {"thing":{"comment":{"type":"bar"}}} # In: {"thing":{"type":"baz"}} ``` Opening brackets on a field begins a query where the context of `this` changes to value of the path it is opened upon, therefore in the above example `this` within the brackets refers to the contents of `this.thing`. ## [](#literals)Literals Bloblang supports number, boolean, string, null, array and object literals: ```bloblang root = [ 7, false, "string", null, { "first": 11, "second": {"foo":"bar"}, "third": """multiple lines on this string""" } ] # In: {} ``` The values within literal arrays and objects can be dynamic query expressions, as well as the keys of object literals. ## [](#comments)Comments You might’ve already spotted, comments are started with a hash (`#`) and end with a line break: ```bloblang root = this.some.value # And now this is a comment ``` ## [](#boolean-logic-and-arithmetic)Boolean logic and arithmetic Bloblang supports a range of boolean operators `!`, `>`, `>=`, `==`, `<`, `<=`, `&&`, `||` and mathematical operators `+`, `-`, `*`, `/`, `%`: ```bloblang root.is_big = this.number > 100 root.multiplied = this.number * 7 # In: {"number":50} # In: {"number":150} ``` For more information about these operators and how they work check out [the arithmetic page](../arithmetic/). ## [](#conditional-mapping)Conditional mapping Use `if` as either a statement or an expression in order to perform maps conditionally: ```bloblang root = this root.sorted_foo = if this.foo.type() == "array" { this.foo.sort() } if this.foo.type() == "string" { root.upper_foo = this.foo.uppercase() root.lower_foo = this.foo.lowercase() } # In: {"foo":"FooBar"} # In: {"foo":["foo","bar"]} ``` And add as many `else if` queries as you like, followed by an optional final fallback `else`: ```bloblang root.sound = if this.type == "cat" { this.cat.meow } else if this.type == "dog" { this.dog.woof.uppercase() } else { "sweet sweet silence" } # In: {"type":"cat","cat":{"meow":"meeeeooooow!"}} # In: {"type":"dog","dog":{"woof":"guurrrr woof woof!"}} # In: {"type":"caterpillar","caterpillar":{"name":"oleg"}} ``` ## [](#pattern-matching)Pattern matching A `match` expression allows you to perform conditional mappings on a value, each case should be either a boolean expression, a literal value to compare against the target value, or an underscore (`_`) which captures values that have not matched a prior case: ```bloblang root.new_doc = match this.doc { this.type == "article" => this.article this.type == "comment" => this.comment _ => this } # In: {"doc":{"type":"article","article":{"id":"foo","content":"qux"}}} # In: {"doc":{"type":"comment","comment":{"id":"bar","content":"quz"}}} # In: {"doc":{"type":"neither","content":"some other stuff unchanged"}} ``` Within a match block the context of `this` changes to the pattern matched expression, therefore `this` within the match expression above refers to `this.doc`. Match cases can specify a literal value for simple comparison: ```bloblang root = this root.type = match this.type { "doc" => "document", "art" => "article", _ => this } # In: {"type":"doc","foo":"bar"} ``` The match expression can also be left unset which means the context remains unchanged, and the catch-all case can also be omitted: ```bloblang root.new_doc = match { this.doc.type == "article" => this.doc.article this.doc.type == "comment" => this.doc.comment } # In: {"doc":{"type":"neither","content":"some other stuff unchanged"}} ``` If no case matches then the mapping is skipped entirely, hence we would end up with the original document in this case. ## [](#functions)Functions Functions can be placed anywhere and allow you to extract information from your environment, generate values, or access data from the underlying message being mapped: ```bloblang root.doc.id = uuid_v4() root.doc.received_at = now() root.doc.host = hostname() ``` Functions support both named and nameless style arguments: ```bloblang root.values_one = range(start: 0, stop: this.max, step: 2) root.values_two = range(0, this.max, 2) # In: {"max":10} ``` You can find a full list of functions and their parameters in [the functions page](../functions/). ## [](#methods)Methods Methods are similar to functions but enact upon a target value, these provide most of the power in Bloblang as they allow you to augment query values and can be added to any expression (including other methods): ```bloblang root.doc.id = this.thing.id.string().catch(uuid_v4()) root.doc.reduced_nums = this.thing.nums.map_each(num -> if num < 10 { deleted() } else { num - 10 }) root.has_good_taste = ["pikachu","mewtwo","magmar"].contains(this.user.fav_pokemon) # In: {"thing":{"id":123,"nums":[5,12,8,15,20]},"user":{"fav_pokemon":"pikachu"}} ``` Methods also support both named and nameless style arguments: ```bloblang root.foo_one = this.(bar | baz).trim().replace_all(old: "dog", new: "cat") root.foo_two = this.(bar | baz).trim().replace_all("dog", "cat") # In: {"bar":" I love my dog "} ``` You can find a full list of methods and their parameters in [the methods page](../methods/). ## [](#maps)Maps Defining named maps allows you to reuse common mappings on values with the [`apply` method](../methods/#apply): ```bloblang map things { root.first = this.thing_one root.second = this.thing_two } root.foo = this.value_one.apply("things") root.bar = this.value_two.apply("things") # In: {"value_one":{"thing_one":"hey","thing_two":"yo"},"value_two":{"thing_one":"sup","thing_two":"waddup"}} ``` Within a map the keyword `root` refers to a newly created document that will replace the target of the map, and `this` refers to the original value of the target. The argument of `apply` is a string, which allows you to dynamically resolve the mapping to apply. ## [](#import-maps)Import maps It’s possible to import maps defined in a file with an `import` statement: ```bloblang import "./common_maps.blobl" root.foo = this.value_one.apply("things") root.bar = this.value_two.apply("things") # In: {"value_one":{"thing_one":"hey","thing_two":"yo"},"value_two":{"thing_one":"sup","thing_two":"waddup"}} ``` Imports from a Bloblang mapping within a Redpanda Connect config are relative to the process running the config. Imports from an imported file are relative to the file that is importing it. ## [](#filtering)Filtering By assigning the root of a mapped document to the `deleted()` function you can delete a message entirely: ```bloblang # Filter all messages that have fewer than 10 URLs. root = if this.doc.urls.length() < 10 { deleted() } # In: {"doc":{"urls":["a","b","c"]}} # In: {"doc":{"urls":["a","b","c","d","e","f","g","h","i","j"]}} ``` ## [](#error-handling)Error handling Functions and methods can fail under certain circumstances, such as when they receive types they aren’t able to act upon. These failures, when not caught, will cause the entire mapping to fail. However, the [method `catch`](../methods/#catch) can be used in order to return a value when a failure occurs instead: ```bloblang # Map an empty array to `foo` if the field `bar` is not a string. root.foo = this.bar.split(",").catch([]) # In: {"bar":"a,b,c"} # In: {"bar":123} ``` Since `catch` is a method it can also be attached to bracketed map expressions: ```bloblang # Map `false` if any of the operations in this boolean query fail. root.thing = ( this.foo > this.bar && this.baz.contains("wut") ).catch(false) # In: {"foo":10,"bar":5,"baz":"wut wut"} # In: {"foo":"not a number","bar":5,"baz":"wut wut"} ``` And one of the more powerful features of Bloblang is that a single `catch` method at the end of a chain of methods can recover errors from any method in the chain: ```bloblang # Catch errors caused by: # - foo not existing # - foo not being a string # - an element from split foo not being a valid JSON string root.things = this.foo.split(",").map_each( ele -> ele.parse_json() ).catch([]) # Specifically catch a JSON parse error root.things = this.foo.split(",").map_each( ele -> ele.parse_json().catch({}) ) # In: {"foo":"{\"a\":1},{\"b\":2}"} # In: {"foo":"not valid json"} ``` However, the `catch` method only acts on errors, sometimes it’s also useful to set a fall back value when a query returns `null` in which case the [method `or`](../methods/#or) can be used the same way: ```bloblang # Map "default" if either the element index 5 does not exist, or the underlying # element is `null`. root.foo = this.bar.index(5).or("default") # In: {"bar":["a","b","c"]} # In: {"bar":["a","b","c","d","e","f","g"]} ``` ## [](#unit-testing)Unit testing It’s possible to execute unit tests for your Bloblang mappings using the standard Redpanda Connect unit test capabilities outlined [in this document](../../../configuration/unit_testing/). ## [](#troubleshooting)Troubleshooting 1. I’m seeing `unable to reference message as structured (with 'this')` when I try to run mappings with `rpk connect blobl`. That particular error message means the mapping is failing to parse what’s being fed in as a JSON document. Make sure that the data you are feeding in is valid JSON, and also that the documents _do not_ contain line breaks as `rpk connect blobl` will parse each line individually. Why? That’s a good question. Bloblang supports non-JSON formats too, so it can’t delimit documents with a streaming JSON parser like tools such as `jq`, so instead it uses line breaks to determine the boundaries of each message. --- # Page 366: Advanced Bloblang **URL**: https://docs.redpanda.com/redpanda-connect/guides/bloblang/advanced.md --- # Advanced Bloblang --- title: Advanced Bloblang latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: bloblang/advanced page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: bloblang/advanced.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/bloblang/advanced.adoc description: Some advanced Bloblang patterns page-git-created-date: "2024-05-24" page-git-modified-date: "2026-02-04" --- ## [](#map-parameters)Map parameters A map definition only has one input parameter, which is the context that it is called upon: ```bloblang map formatting { root = "(%v)".format(this) } root.a = this.a.apply("formatting") root.b = this.b.apply("formatting") # In: {"a":"foo","b":"bar"} ``` However, we can use object literals in order to provide multiple map parameters. Imagine if we wanted a map that is the exact same as above except the pattern is `[%v]` instead, with the potential for even more patterns in the future. To do that we can pass an object with a field `value` with our target to map and a field `pattern` which allows us to specify the pattern to apply: ```bloblang map formatting { root = this.pattern.format(this.value) } root.a = { "value":this.a, "pattern":this.pattern, }.apply("formatting") root.b = { "value":this.b, "pattern":this.pattern, }.apply("formatting") # In: {"a":"foo","b":"bar","pattern":"[%v]"} ``` ## [](#walking-the-tree)Walking the tree Sometimes it’s necessary to perform a mapping on all values within an unknown tree structure. You can do that easily with recursive mapping: ```bloblang map unescape_values { root = match { this.type() == "object" => this.map_each(item -> item.value.apply("unescape_values")), this.type() == "array" => this.map_each(ele -> ele.apply("unescape_values")), this.type() == "string" => this.unescape_html(), this.type() == "bytes" => this.unescape_html(), _ => this, } } root = this.apply("unescape_values") # In: {"first":{"nested":"foo & bar"},"second":10,"third":["1 < 2",{"also_nested":"2 > 1"}]} ``` ## [](#message-expansion)Message expansion Expanding a single message into multiple messages can be done by mapping messages into an array and following it up with an [`unarchive` processor](../../../components/processors/unarchive/). For example, given documents of this format: ```json { "id": "foobar", "items": [ {"content":"foo"}, {"content":"bar"}, {"content":"baz"} ] } ``` We can pull `items` out to the root with `root = items` with a [`mapping` processor](../../../components/processors/mapping/) and follow it with an [`unarchive` processor](../../../components/processors/unarchive/) to expand each element into its own independent message: ```yaml pipeline: processors: - mapping: root = this.items - unarchive: format: json_array ``` However, most of the time we also need to map the elements before expanding them, and often that includes copying fields outside of our target array. We can do that with methods such as `map_each` and `merge`: ```bloblang # In: {"id":"foobar","items":[{"content":"foo"},{"content":"bar"},{"content":"baz"}]} root = this.items.map_each(ele -> this.without("items").merge(ele)) ``` However, the above mapping is slightly inefficient as we would create a copy of our source object for each element with the `this.without("items")` part. A more efficient way to do this would be to capture that query within a variable: ```bloblang # In: {"id":"foobar","items":[{"content":"foo"},{"content":"bar"},{"content":"baz"}]} let doc_root = this.without("items") root = this.items.map_each($doc_root.merge(this)) ``` Also note that when we set `doc_root` we remove the field `items` from the target document. The full config would now be: ```yaml pipeline: processors: - mapping: | let doc_root = this.without("items") root = this.items.map_each($doc_root.merge(this)) - unarchive: format: json_array ``` ## [](#creating-csv)Creating CSV Redpanda Connect has a few different ways of outputting a stream of CSV data. However, the best way to do it is by converting the documents into CSV rows with Bloblang as this gives you full control over exactly how the schema is generated, erroneous data is handled, and escaping of column data is performed. A common and simple use case is to simply flatten documents and write out the column values in alphabetical order. The first row we generate should also be prefixed with a row containing those column names. Here’s a mapping that achieves this by using a `count` function to detect the very first invocation of the mapping in a stream pipeline: ```bloblang # In: {"name":"foo","address":"123 Main St, Apt 2"} # foo,"123 Main St, Apt 2" map escape_csv { root = if this.re_match("[\"\n,]+") { "\"" + this.replace_all("\"", "\"\"") + "\"" } else { this } } # Extract key/value pairs as an array and sort by the key let kvs = this.key_values().sort_by(v -> v.key) # Create a header prefix for our output only on the first row let header = if count("rows_in_file") == 1 { $kvs.map_each(kv -> kv.key.apply("escape_csv")).join(",") + "\n" } else { "" } root = $header + $kvs.map_each(kv -> kv.value.string().apply("escape_csv")).join(",") ``` And with this mapping we can write the data to a newly created CSV file using an output with a simple `lines` codec: ```yaml output: file: path: ./result.csv codec: lines ``` Perhaps the first expansion of this mapping that would be worthwhile is to add an explicit list of column names, or at least confirm that the number of values in a row matches an expected count. --- # Page 367: Bloblang Arithmetic **URL**: https://docs.redpanda.com/redpanda-connect/guides/bloblang/arithmetic.md --- # Bloblang Arithmetic --- title: Bloblang Arithmetic latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: bloblang/arithmetic page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: bloblang/arithmetic.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/bloblang/arithmetic.adoc description: How arithmetic works within Bloblang page-git-created-date: "2024-05-24" page-git-modified-date: "2024-11-25" --- Bloblang supports a range of comparison operators `!`, `>`, `>=`, `==`, `<`, `<=`, `&&`, `||` and mathematical operators `+`, `-`, `*`, `/`, `%`. How these operators behave is dependent on the type of the values they’re used with, and therefore it’s worth fully understanding these behaviors if you intend to use them heavily in your mappings. ## [](#mathematical)Mathematical All mathematical operators (`+`, `-`, `*`, `/`, `%`) are valid against number values, and addition (`+`) is also supported when both the left and right hand side arguments are strings. If a mathematical operator is used with an argument that is non-numeric (with the aforementioned string exception) then a [recoverable mapping error will be thrown](../about/#error-handling). ### [](#number-degradation)Number degradation In Bloblang any number resulting from a method, function or arithmetic is either a 64-bit signed integer or a 64-bit floating point value. Numbers from input documents can be any combination of size and be signed or unsigned. When a mathematical operation is performed with two or more integer values Bloblang will create an integer result, with the exception of division. However, if any number within a mathematical operation is a floating point then the result will be a floating point value. In order to explicitly coerce numbers into integer types you can use the [`.ceil()`, `.floor()`, or `.round()` methods](../methods/#number-manipulation). ## [](#comparison)Comparison The not (`!`) operator reverses the boolean value of the expression immediately following it, and is valid to place before any query that yields a boolean value. If the following expression yields a non-boolean value then a [recoverable mapping error will be thrown](../about/#error-handling). If you wish to reverse the boolean result of a complex query then simply place the query within brackets (`!(this.foo > this.bar)`). ### [](#equality)Equality The equality operators (`==` and `!=`) are valid to use against any value type. In order for arguments to be considered equal they must match in both their basic type (`string`, `number`, `null`, `bool`, etc) as well as their value. If you wish to compare mismatched value types then use [coercion methods](../methods/#type-coercion). Number arguments are considered equal if their value is the same when represented the same way, which means their underlying representations (integer, float, etc) do not need to match in order for them to be considered equal. ### [](#numerical)Numerical Numerical comparisons (`>`, `>=`, `<`, `<=`) are valid to use against number values only. If a non-number value is used as an argument then a [recoverable mapping error will be thrown](../about/#error-handling). ### [](#boolean)Boolean Boolean comparison operators (`||`, `&&`) are valid to use against boolean values only (`true` or `false`). If a non-boolean value is used as an argument then a [recoverable mapping error will be thrown](../about/#error-handling). --- # Page 368: Bloblang Functions **URL**: https://docs.redpanda.com/redpanda-connect/guides/bloblang/functions.md --- # Bloblang Functions --- title: Bloblang Functions latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: bloblang/functions page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: bloblang/functions.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/bloblang/functions.adoc description: A list of Bloblang functions page-git-created-date: "2024-05-24" page-git-modified-date: "2026-04-02" --- Functions can be placed anywhere and allow you to extract information from your environment, generate values, or access data from the underlying message being mapped: ```bloblang root.doc.id = uuid_v4() root.doc.received_at = now() root.doc.host = hostname() ``` Functions support both named and nameless style arguments: ```bloblang root.values_one = range(start: 0, stop: this.max, step: 2) root.values_two = range(0, this.max, 2) # In: {"max":10} ``` ## [](#batch_index)batch\_index Returns the zero-based index of the current message within its batch. Use this to conditionally process messages based on their position, or to create sequential identifiers within a batch. ### [](#examples)Examples ```bloblang root = if batch_index() > 0 { deleted() } ``` Create a unique identifier combining batch position with timestamp: ```bloblang root.id = "%v-%v".format(timestamp_unix(), batch_index()) ``` ## [](#batch_size)batch\_size Returns the total number of messages in the current batch. Use this to determine batch boundaries or compute relative positions. ### [](#examples-2)Examples ```bloblang root.total = batch_size() ``` Check if processing the last message in a batch: ```bloblang root.is_last = batch_index() == batch_size() - 1 ``` ## [](#bytes)bytes Creates a zero-initialized byte array of specified length. Use this to allocate fixed-size byte buffers for binary data manipulation or to generate padding. ### [](#parameters)Parameters | Name | Type | Description | | --- | --- | --- | | length | integer | The size of the resulting byte array. | ### [](#examples-3)Examples ```bloblang root.data = bytes(5) ``` Create a buffer for binary operations: ```bloblang root.header = bytes(16) root.payload = content() ``` ## [](#content)content Returns the raw message payload as bytes, regardless of the current mapping context. Use this to access the original message when working within nested contexts, or to store the entire message as a field. ### [](#examples-4)Examples ```bloblang root.doc = content().string() # In: {"foo":"bar"} # Out: {"doc":"{\"foo\":\"bar\"}"} ``` Preserve original message while adding metadata: ```bloblang root.original = content().string() root.processed_by = "ai" # In: {"foo":"bar"} # Out: {"original":"{\"foo\":\"bar\"}","processed_by":"ai"} ``` ## [](#count)count > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. The `count` function is a counter starting at 1 which increments after each time it is called. Count takes an argument which is an identifier for the counter, allowing you to specify multiple unique counters in your configuration. ### [](#parameters-2)Parameters | Name | Type | Description | | --- | --- | --- | | name | string | An identifier for the counter. | ### [](#examples-5)Examples ```bloblang root = this root.id = count("bloblang_function_example") # In: {"message":"foo"} # Out: {"id":1,"message":"foo"} # In: {"message":"bar"} # Out: {"id":2,"message":"bar"} ``` ## [](#counter)counter Generates an incrementing sequence of integers starting from a minimum value (default 1). Each counter instance maintains its own independent state across message processing. When the maximum value is reached, the counter automatically resets to the minimum. ### [](#parameters-3)Parameters | Name | Type | Description | | --- | --- | --- | | min | query expression | The starting value of the counter. This is the first value yielded. Evaluated once when the mapping is initialized. | | max | query expression | The maximum value before the counter resets to min. Evaluated once when the mapping is initialized. | | set (optional) | query expression | An optional query that controls counter behavior: when it resolves to a non-negative integer, the counter is set to that value; when it resolves to null, the counter is read without incrementing; when it resolves to a deletion, the counter resets to min; otherwise the counter increments normally. | ### [](#examples-6)Examples Generate sequential IDs for each message: ```bloblang root.id = counter() # In: {} # Out: {"id":1} # In: {} # Out: {"id":2} ``` Use a custom range for the counter: ```bloblang root.batch_num = counter(min: 100, max: 200) # In: {} # Out: {"batch_num":100} # In: {} # Out: {"batch_num":101} ``` Increment a counter multiple times within a single mapping using a named map: ```bloblang map increment { root = counter() } root.first_id = null.apply("increment") root.second_id = null.apply("increment") # In: {} # Out: {"first_id":1,"second_id":2} # In: {} # Out: {"first_id":3,"second_id":4} ``` Conditionally reset a counter based on input data: ```bloblang root.streak = counter(set: if this.status != "success" { 0 }) # In: {"status":"success"} # Out: {"streak":1} # In: {"status":"success"} # Out: {"streak":2} # In: {"status":"failure"} # Out: {"streak":0} # In: {"status":"success"} # Out: {"streak":1} ``` Peek at the current counter value without incrementing by using null in the set parameter: ```bloblang root.count = counter(set: if this.peek { null }) # In: {"peek":false} # Out: {"count":1} # In: {"peek":false} # Out: {"count":2} # In: {"peek":true} # Out: {"count":2} # In: {"peek":false} # Out: {"count":3} ``` ## [](#deleted)deleted Returns a deletion marker that removes the target field or message. When applied to root, the entire message is dropped while still being acknowledged as successfully processed. Use this to filter data or conditionally remove fields. ### [](#examples-7)Examples ```bloblang root = this root.bar = deleted() # In: {"bar":"bar_value","baz":"baz_value","foo":"foo value"} # Out: {"baz":"baz_value","foo":"foo value"} ``` Filter array elements by returning deleted for unwanted items: ```bloblang root.new_nums = this.nums.map_each(num -> if num < 10 { deleted() } else { num - 10 }) # In: {"nums":[3,11,4,17]} # Out: {"new_nums":[1,7]} ``` ## [](#env)env Reads an environment variable and returns its value as a string. Returns `null` if the variable is not set. By default, values are cached for performance. ### [](#parameters-4)Parameters | Name | Type | Description | | --- | --- | --- | | name | string | The name of the environment variable to read. | | no_cache | bool | Disable caching to read the latest value on each invocation. | ### [](#examples-8)Examples ```bloblang root.api_key = env("API_KEY") ``` ```bloblang root.database_url = env("DB_URL").or("localhost:5432") ``` Use `no_cache` to read updated environment variables during runtime, useful for dynamic configuration changes: ```bloblang root.config = env(name: "DYNAMIC_CONFIG", no_cache: true) ``` ## [](#error)error Returns the error message string if the message has failed processing, otherwise `null`. Use this in error handling pipelines to log or route failed messages based on their error details. ### [](#examples-9)Examples ```bloblang root.doc.error = error() ``` Route messages to different outputs based on error presence: ```bloblang root = this root.error_msg = error() root.has_error = error() != null ``` ## [](#error_source_label)error\_source\_label Returns the user-defined label of the component that caused the error, empty string if no label is set, or `null` if the message has no error. Use this for more human-readable error tracking when components have custom labels. ### [](#examples-10)Examples ```bloblang root.doc.error_source_label = error_source_label() ``` Route errors based on component labels: ```bloblang root.error_category = error_source_label().or("unknown") ``` ## [](#error_source_name)error\_source\_name Returns the component name that caused the error, or `null` if the message has no error or the error has no associated component. Use this to identify which processor or component in your pipeline caused a failure. ### [](#examples-11)Examples ```bloblang root.doc.error_source_name = error_source_name() ``` Create detailed error logs with component information: ```bloblang root.error_details = if errored() { { "message": error(), "component": error_source_name(), "timestamp": now() } } ``` ## [](#error_source_path)error\_source\_path Returns the dot-separated path to the component that caused the error, or `null` if the message has no error. Use this to identify the exact location of a failed component in nested pipeline configurations. ### [](#examples-12)Examples ```bloblang root.doc.error_source_path = error_source_path() ``` Build comprehensive error context for debugging: ```bloblang root.error_info = { "path": error_source_path(), "component": error_source_name(), "message": error() } ``` ## [](#errored)errored Returns true if the message has failed processing, false otherwise. Use this for conditional logic in error handling workflows or to route failed messages to dead letter queues. ### [](#examples-13)Examples ```bloblang root.doc.status = if errored() { 400 } else { 200 } ``` Send only failed messages to a separate stream: ```bloblang root = if errored() { this } else { deleted() } ``` ## [](#fake)fake Generates realistic fake data for testing and development purposes. Supports a wide variety of data types including personal information, network addresses, dates/times, financial data, and UUIDs. Useful for creating mock data, populating test databases, or anonymizing sensitive information. Supported functions: `latitude`, `longitude`, `unix_time`, `date`, `time_string`, `month_name`, `year_string`, `day_of_week`, `day_of_month`, `timestamp`, `century`, `timezone`, `time_period`, `email`, `mac_address`, `domain_name`, `url`, `username`, `ipv4`, `ipv6`, `password`, `jwt`, `word`, `sentence`, `paragraph`, `cc_type`, `cc_number`, `currency`, `amount_with_currency`, `title_male`, `title_female`, `first_name`, `first_name_male`, `first_name_female`, `last_name`, `name`, `gender`, `chinese_first_name`, `chinese_last_name`, `chinese_name`, `phone_number`, `toll_free_phone_number`, `e164_phone_number`, `uuid_hyphenated`, `uuid_digit`. ### [](#parameters-5)Parameters | Name | Type | Description | | --- | --- | --- | | function | string | The name of the faker function to use. See description for full list of supported functions. | ### [](#examples-14)Examples Generate fake user profile data for testing: ```bloblang root.user = { "id": fake("uuid_hyphenated"), "name": fake("name"), "email": fake("email"), "created_at": fake("timestamp") } ``` Create realistic test data for network monitoring: ```bloblang root.event = { "source_ip": fake("ipv4"), "mac_address": fake("mac_address"), "url": fake("url") } ``` ## [](#file)file Reads a file and returns its contents as bytes. Paths are resolved from the process working directory. For paths relative to the mapping file, use `file_rel`. By default, files are cached after first read. ### [](#parameters-6)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | The absolute or relative path to the file. | | no_cache | bool | Disable caching to read the latest file contents on each invocation. | ### [](#examples-15)Examples ```bloblang root.config = file("/etc/config.json").parse_json() ``` ```bloblang root.template = file("./templates/email.html").string() ``` Use `no_cache` to read updated file contents during runtime, useful for hot-reloading configuration: ```bloblang root.rules = file(path: "/etc/rules.yaml", no_cache: true).parse_yaml() ``` ## [](#file_rel)file\_rel Reads a file and returns its contents as bytes. Paths are resolved relative to the mapping file’s directory, making it portable across different environments. By default, files are cached after first read. ### [](#parameters-7)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | The path to the file, relative to the mapping file’s directory. | | no_cache | bool | Disable caching to read the latest file contents on each invocation. | ### [](#examples-16)Examples ```bloblang root.schema = file_rel("./schemas/user.json").parse_json() ``` ```bloblang root.lookup = file_rel("../data/lookup.csv").parse_csv() ``` Use `no_cache` to read updated file contents during runtime, useful for reloading data files without restarting: ```bloblang root.translations = file_rel(path: "./i18n/en.yaml", no_cache: true).parse_yaml() ``` ## [](#hostname)hostname Returns the hostname of the machine running Benthos. Useful for identifying which instance processed a message in distributed deployments. ### [](#examples-17)Examples ```bloblang root.processed_by = hostname() ``` ## [](#json)json Returns a field from the original JSON message by dot path, always accessing the root document regardless of mapping context. Use this to reference the source message when working in nested contexts or to extract specific fields. ### [](#parameters-8)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | An optional [dot path][field_paths] identifying a field to obtain. | ### [](#examples-18)Examples ```bloblang root.mapped = json("foo.bar") # In: {"foo":{"bar":"hello world"}} # Out: {"mapped":"hello world"} ``` Access the original message from within nested mapping contexts: ```bloblang root.doc = json() # In: {"foo":{"bar":"hello world"}} # Out: {"doc":{"foo":{"bar":"hello world"}}} ``` ## [](#ksuid)ksuid Generates a K-Sortable Unique Identifier with built-in timestamp ordering. Use this for distributed unique IDs that sort chronologically and remain collision-resistant without coordination between generators. ### [](#examples-19)Examples ```bloblang root.id = ksuid() ``` Create sortable event IDs for logging: ```bloblang root.event = { "id": ksuid(), "type": this.event_type, "data": this.payload } ``` ## [](#meta)meta > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Returns the value of a metadata key from the input message as a string, or `null` if the key does not exist. Since values are extracted from the read-only input message they do NOT reflect changes made from within the map. In order to query metadata mutations made within a mapping use the [`root_meta` function](#root_meta). This function supports extracting metadata from other messages of a batch with the `from` method. ### [](#parameters-9)Parameters | Name | Type | Description | | --- | --- | --- | | key | string | An optional key of a metadata value to obtain. | ### [](#examples-20)Examples ```bloblang root.topic = meta("kafka_topic") ``` The key parameter is optional and if omitted the entire metadata contents are returned as an object: ```bloblang root.all_metadata = meta() ``` ## [](#metadata)metadata Returns metadata from the input message by key, or `null` if the key doesn’t exist. This reads the original metadata; to access modified metadata during mapping, use the `@` operator instead. Use this to extract message properties like topics, headers, or timestamps. ### [](#parameters-10)Parameters | Name | Type | Description | | --- | --- | --- | | key | string | An optional key of a metadata value to obtain. | ### [](#examples-21)Examples ```bloblang root.topic = metadata("kafka_topic") ``` Retrieve all metadata as an object by omitting the key parameter: ```bloblang root.all_metadata = metadata() ``` Copy specific metadata fields to the message body: ```bloblang root.source = { "topic": metadata("kafka_topic"), "partition": metadata("kafka_partition"), "timestamp": metadata("kafka_timestamp_unix") } ``` ## [](#nanoid)nanoid Generates a URL-safe unique identifier using Nano ID. Use this for compact, URL-friendly IDs with good collision resistance. Customize the length (default 21) or provide a custom alphabet for specific character requirements. ### [](#parameters-11)Parameters | Name | Type | Description | | --- | --- | --- | | length (optional) | integer | An optional length. | | alphabet (optional) | string | An optional custom alphabet to use for generating IDs. When specified the field length must also be present. | ### [](#examples-22)Examples ```bloblang root.id = nanoid() ``` Generate a longer ID for additional uniqueness: ```bloblang root.id = nanoid(54) ``` Use a custom alphabet for domain-specific IDs: ```bloblang root.id = nanoid(54, "abcde") ``` ## [](#nothing)nothing ## [](#now)now Returns the current timestamp as an RFC 3339 formatted string with nanosecond precision. Use this to add processing timestamps to messages or measure time between events. Chain with `ts_format` to customize the format or timezone. ### [](#examples-23)Examples ```bloblang root.received_at = now() ``` Format the timestamp in a custom format and timezone: ```bloblang root.received_at = now().ts_format("Mon Jan 2 15:04:05 -0700 MST 2006", "UTC") ``` ## [](#pi)pi Returns the value of the mathematical constant Pi. ### [](#examples-24)Examples ```bloblang root.radians = this.degrees * (pi() / 180) # In: {"degrees":45} # Out: {"radians":0.7853981633974483} ``` ```bloblang root.degrees = this.radians * (180 / pi()) # In: {"radians":0.78540} # Out: {"degrees":45.00010522957486} ``` ## [](#random_int)random\_int Generates a pseudo-random non-negative 64-bit integer. Use this for creating random IDs, sampling data, or generating test values. Provide a seed for reproducible randomness, or use a dynamic seed like `timestamp_unix_nano()` for unique values per mapping instance. Optional `min` and `max` parameters constrain the output range (both inclusive). For dynamic ranges based on message data, use the modulo operator instead: `random_int() % dynamic_max + dynamic_min`. ### [](#parameters-12)Parameters | Name | Type | Description | | --- | --- | --- | | seed | query expression | A seed to use, if a query is provided it will only be resolved once during the lifetime of the mapping. | | min | integer | The minimum value the random generated number will have. The default value is 0. | | max | integer | The maximum value the random generated number will have. The default value is 9223372036854775806 (math.MaxInt64 - 1). | ### [](#examples-25)Examples ```bloblang root.first = random_int() root.second = random_int(1) root.third = random_int(max:20) root.fourth = random_int(min:10, max:20) root.fifth = random_int(timestamp_unix_nano(), 5, 20) root.sixth = random_int(seed:timestamp_unix_nano(), max:20) ``` Use a dynamic seed for unique random values per mapping instance: ```bloblang root.random_id = random_int(timestamp_unix_nano()) root.sample_percent = random_int(seed: timestamp_unix_nano(), min: 0, max: 100) ``` ## [](#range)range Creates an array of integers from start (inclusive) to stop (exclusive) with an optional step. Use this to generate sequences for iteration, indexing, or creating numbered lists. ### [](#parameters-13)Parameters | Name | Type | Description | | --- | --- | --- | | start | integer | The start value. | | stop | integer | The stop value. | | step | integer | The step value. | ### [](#examples-26)Examples ```bloblang root.a = range(0, 10) root.b = range(start: 0, stop: this.max, step: 2) # Using named params root.c = range(0, -this.max, -2) # In: {"max":10} # Out: {"a":[0,1,2,3,4,5,6,7,8,9],"b":[0,2,4,6,8],"c":[0,-2,-4,-6,-8]} ``` Generate a sequence for batch processing: ```bloblang root.pages = range(0, this.total_items, 100).map_each(offset -> { "offset": offset, "limit": 100 }) # In: {"total_items":250} # Out: {"pages":[{"limit":100,"offset":0},{"limit":100,"offset":100}]} ``` ## [](#root_meta)root\_meta > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Returns the value of a metadata key from the new message being created as a string, or `null` if the key does not exist. Changes made to metadata during a mapping will be reflected by this function. ### [](#parameters-14)Parameters | Name | Type | Description | | --- | --- | --- | | key | string | An optional key of a metadata value to obtain. | ### [](#examples-27)Examples ```bloblang root.topic = root_meta("kafka_topic") ``` The key parameter is optional and if omitted the entire metadata contents are returned as an object: ```bloblang root.all_metadata = root_meta() ``` ## [](#snowflake_id)snowflake\_id Generates a unique, time-ordered Snowflake ID. Snowflake IDs are 64-bit integers that encode timestamp, node ID, and sequence information, making them ideal for distributed systems where sortable unique identifiers are needed. Returns a string representation of the ID. ### [](#parameters-15)Parameters | Name | Type | Description | | --- | --- | --- | | node_id | integer | Optional node identifier (0-1023) to distinguish IDs generated by different machines in a distributed system. Defaults to 1. | ### [](#examples-28)Examples Generate a unique Snowflake ID for each message: ```bloblang root.id = snowflake_id() root.payload = this ``` Generate Snowflake IDs with different node IDs for multi-datacenter deployments: ```bloblang root.id = snowflake_id(42) root.data = this ``` ## [](#throw)throw Immediately fails the mapping with a custom error message. Use this to halt processing when data validation fails or required fields are missing, causing the message to be routed to error handlers. ### [](#parameters-16)Parameters | Name | Type | Description | | --- | --- | --- | | why | string | A string explanation for why an error was thrown, this will be added to the resulting error message. | ### [](#examples-29)Examples ```bloblang root.doc.type = match { this.exists("header.id") => "foo" this.exists("body.data") => "bar" _ => throw("unknown type") } root.doc.contents = (this.body.content | this.thing.body) # In: {"header":{"id":"first"},"thing":{"body":"hello world"}} # Out: {"doc":{"contents":"hello world","type":"foo"}} # In: {"nothing":"matches"} # Out: Error("failed assignment (line 1): unknown type") ``` Validate required fields before processing: ```bloblang root = if this.exists("user_id") { this } else { throw("missing required field: user_id") } # In: {"user_id":123,"name":"alice"} # Out: {"name":"alice","user_id":123} # In: {"name":"bob"} # Out: Error("failed assignment (line 1): missing required field: user_id") ``` ## [](#timestamp_unix)timestamp\_unix Returns the current Unix timestamp in seconds since epoch. Use this for numeric timestamps compatible with most systems, or as a seed for random number generation. ### [](#examples-30)Examples ```bloblang root.received_at = timestamp_unix() ``` Create a sortable ID combining timestamp with a counter: ```bloblang root.id = "%v-%v".format(timestamp_unix(), batch_index()) ``` ## [](#timestamp_unix_micro)timestamp\_unix\_micro Returns the current Unix timestamp in microseconds since epoch. Use this for high-precision timing measurements or when microsecond resolution is required. ### [](#examples-31)Examples ```bloblang root.received_at = timestamp_unix_micro() ``` Measure elapsed time between events: ```bloblang root.processing_duration_us = timestamp_unix_micro() - this.start_time_us ``` ## [](#timestamp_unix_milli)timestamp\_unix\_milli Returns the current Unix timestamp in milliseconds since epoch. Use this for millisecond-precision timestamps common in web APIs and JavaScript systems. ### [](#examples-32)Examples ```bloblang root.received_at = timestamp_unix_milli() ``` Add processing time metadata: ```bloblang meta processing_time_ms = timestamp_unix_milli() ``` ## [](#timestamp_unix_nano)timestamp\_unix\_nano Returns the current Unix timestamp in nanoseconds since epoch. Use this for the highest precision timing or as a unique seed value that changes on every invocation. ### [](#examples-33)Examples ```bloblang root.received_at = timestamp_unix_nano() ``` Generate unique random values on each mapping: ```bloblang root.random_value = random_int(timestamp_unix_nano()) ``` ## [](#tracing_id)tracing\_id Returns the OpenTelemetry trace ID for the message, or an empty string if no tracing span exists. Use this to correlate logs and events with distributed traces. ### [](#examples-34)Examples ```bloblang meta trace_id = tracing_id() ``` Add trace ID to structured logs: ```bloblang root.log_entry = this root.log_entry.trace_id = tracing_id() ``` ## [](#tracing_span)tracing\_span Returns the OpenTelemetry tracing span attached to the message as a text map object, or `null` if no span exists. Use this to propagate trace context to downstream systems via headers or metadata. ### [](#examples-35)Examples ```bloblang root.headers.traceparent = tracing_span().traceparent # In: {"some_stuff":"just can't be explained by science"} # Out: {"headers":{"traceparent":"00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"}} ``` Forward all tracing fields to output metadata: ```bloblang meta = tracing_span() ``` ## [](#ulid)ulid Generates a Universally Unique Lexicographically Sortable Identifier (ULID). ULIDs are 128-bit identifiers that are sortable by creation time, URL-safe, and case-insensitive. They consist of a 48-bit timestamp (millisecond precision) and 80 bits of randomness, making them ideal for distributed systems that need time-ordered unique IDs without coordination. ### [](#parameters-17)Parameters | Name | Type | Description | | --- | --- | --- | | encoding | string | Encoding format for the ULID. "crockford" produces 26-character Base32 strings (recommended). "hex" produces 32-character hexadecimal strings. | | random_source | string | Randomness source: "secure_random" uses cryptographically secure random (recommended for production), "fast_random" uses faster but non-secure random (only for non-sensitive testing). | ### [](#examples-36)Examples Generate time-sortable IDs for distributed message ordering: ```bloblang root.message_id = ulid() root.timestamp = now() root.data = this ``` Generate hex-encoded ULIDs for systems that prefer hexadecimal format: ```bloblang root.id = ulid("hex") ``` ## [](#uuid_v4)uuid\_v4 Generates a random RFC-4122 version 4 UUID. Use this for creating unique identifiers that don’t reveal timing information or require ordering. Each invocation produces a new globally unique ID. ### [](#examples-37)Examples ```bloblang root.id = uuid_v4() ``` Add unique request IDs for tracing: ```bloblang root = this root.request_id = uuid_v4() ``` ## [](#uuid_v7)uuid\_v7 Generates a time-ordered UUID version 7 with millisecond timestamp precision. Use this for sortable unique identifiers that maintain chronological ordering, ideal for database keys or event IDs. Optionally specify a custom timestamp. ### [](#parameters-18)Parameters | Name | Type | Description | | --- | --- | --- | | time (optional) | timestamp | An optional timestamp to use for the time ordered portion of the UUID. | ### [](#examples-38)Examples ```bloblang root.id = uuid_v7() ``` Generate a UUID with a specific timestamp for backdating events: ```bloblang root.id = uuid_v7(now().ts_sub_iso8601("PT1M")) ``` ## [](#var)var ### [](#parameters-19)Parameters | Name | Type | Description | | --- | --- | --- | | name | string | The name of the target variable. | ## [](#with_schema_registry_header)with\_schema\_registry\_header Prepends a Confluent Schema Registry wire format header to message bytes. The header is 5 bytes: a magic byte (0x00) followed by a 4-byte big-endian schema ID. This format is required when producing messages to Kafka topics that use Confluent Schema Registry for schema validation and evolution. ### [](#parameters-20)Parameters | Name | Type | Description | | --- | --- | --- | | schema_id | unknown | The schema ID from your Schema Registry (0 to 4294967295). This ID references the schema version used to encode the message. | | message | unknown | The serialized message bytes (e.g., Avro, Protobuf, or JSON Schema encoded data) to prepend the header to. | ### [](#examples-39)Examples Add Schema Registry header to Avro-encoded message: ```bloblang root = with_schema_registry_header(123, content()) ``` Use schema ID from metadata to add header dynamically: ```bloblang root = with_schema_registry_header(meta("schema_id").number(), content()) ``` --- # Page 369: Bloblang Methods **URL**: https://docs.redpanda.com/redpanda-connect/guides/bloblang/methods.md --- # Bloblang Methods --- title: Bloblang Methods latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: bloblang/methods page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: bloblang/methods.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/bloblang/methods.adoc description: A list of Bloblang methods page-git-created-date: "2024-05-24" page-git-modified-date: "2026-04-02" --- Methods provide most of the power in Bloblang as they allow you to augment values and can be added to any expression (including other methods): ```bloblang root.doc.id = this.thing.id.string().catch(uuid_v4()) root.doc.reduced_nums = this.thing.nums.map_each(num -> if num < 10 { deleted() } else { num - 10 }) root.has_good_taste = ["pikachu","mewtwo","magmar"].contains(this.user.fav_pokemon) # In: {"thing":{"id":123,"nums":[5,12,18,7,25]},"user":{"fav_pokemon":"pikachu"}} ``` Methods support both named and nameless style arguments: ```bloblang root.foo_one = this.(bar | baz).trim().replace_all(old: "dog", new: "cat") root.foo_two = this.(bar | baz).trim().replace_all("dog", "cat") # In: {"bar":" I love my dog "} ``` ## [](#general)General ### [](#apply)apply Apply a declared mapping to a target value. #### [](#parameters)Parameters | Name | Type | Description | | --- | --- | --- | | mapping | string | The mapping to apply. | #### [](#examples)Examples ```bloblang map thing { root.inner = this.first } root.foo = this.doc.apply("thing") # In: {"doc":{"first":"hello world"}} # Out: {"foo":{"inner":"hello world"}} ``` ```bloblang map create_foo { root.name = "a foo" root.purpose = "to be a foo" } root = this root.foo = null.apply("create_foo") # In: {"id":"1234"} # Out: {"foo":{"name":"a foo","purpose":"to be a foo"},"id":"1234"} ``` ### [](#catch)catch If the result of a target query fails (due to incorrect types, failed parsing, etc) the argument is returned instead. #### [](#parameters-2)Parameters | Name | Type | Description | | --- | --- | --- | | fallback | query expression | A value to yield, or query to execute, if the target query fails. | #### [](#examples-2)Examples ```bloblang root.doc.id = this.thing.id.string().catch(uuid_v4()) ``` The fallback argument can be a mapping, allowing you to capture the error string and yield structured data back: ```bloblang root.url = this.url.parse_url().catch(err -> {"error":err,"input":this.url}) # In: {"url":"invalid %&# url"} # Out: {"url":{"error":"field `this.url`: parse \"invalid %&\": invalid URL escape \"%&\"","input":"invalid %&# url"}} ``` When the input document is not structured attempting to reference structured fields with `this` will result in an error. Therefore, a convenient way to delete non-structured data is with a catch: ```bloblang root = this.catch(deleted()) # In: {"doc":{"foo":"bar"}} # Out: {"doc":{"foo":"bar"}} # In: not structured data # Out: ``` ### [](#from)from Modifies a target query such that certain functions are executed from the perspective of another message in the batch. This allows you to mutate events based on the contents of other messages. Functions that support this behavior are `content`, `json` and `meta`. #### [](#parameters-3)Parameters | Name | Type | Description | | --- | --- | --- | | index | integer | The message index to use as a perspective. | #### [](#examples-3)Examples For example, the following map extracts the contents of the JSON field `foo` specifically from message index `1` of a batch, effectively overriding the field `foo` for all messages of a batch to that of message 1: ```bloblang root = this root.foo = json("foo").from(1) ``` ### [](#from_all)from\_all Modifies a target query such that certain functions are executed from the perspective of each message in the batch, and returns the set of results as an array. Functions that support this behavior are `content`, `json` and `meta`. #### [](#examples-4)Examples ```bloblang root = this root.foo_summed = json("foo").from_all().sum() ``` ### [](#map)map Executes a query on the target value, allowing you to transform or extract data from the current context. #### [](#parameters-4)Parameters | Name | Type | Description | | --- | --- | --- | | query | query expression | A query to execute on the target. | ### [](#not)not Returns the logical NOT (negation) of a boolean value. Converts true to false and false to true. ### [](#or)or If the result of the target query fails or resolves to `null`, returns the argument instead. This is an explicit method alternative to the coalesce pipe operator `|`. #### [](#parameters-5)Parameters | Name | Type | Description | | --- | --- | --- | | fallback | query expression | A value to yield, or query to execute, if the target query fails or resolves to null. | #### [](#examples-5)Examples ```bloblang root.doc.id = this.thing.id.or(uuid_v4()) ``` ## [](#encoding-and-encryption)Encoding and encryption ### [](#compress)compress Compresses a string or byte array using the specified compression algorithm. Returns compressed data as bytes. Useful for reducing payload size before transmission or storage. #### [](#parameters-6)Parameters | Name | Type | Description | | --- | --- | --- | | algorithm | string | The compression algorithm: flate, gzip, pgzip (parallel gzip), lz4, snappy, zlib, or zstd. | | level | integer | Compression level (default: -1 for default compression). Higher values increase compression ratio but use more CPU. Range and effect varies by algorithm. | #### [](#examples-6)Examples Compress and encode for safe transmission: ```bloblang root.compressed = content().bytes().compress("gzip").encode("base64") # In: {"message":"hello world I love space"} # Out: {"compressed":"H4sIAAAJbogA/wAmANn/eyJtZXNzYWdlIjoiaGVsbG8gd29ybGQgSSBsb3ZlIHNwYWNlIn0DAHEvdwomAAAA"} ``` Compare compression ratios across algorithms: ```bloblang root.original_size = content().length() root.gzip_size = content().compress("gzip").length() root.lz4_size = content().compress("lz4").length() # In: The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. # Out: {"gzip_size":114,"lz4_size":85,"original_size":89} ``` ### [](#decode)decode Decodes an encoded string according to a chosen scheme. #### [](#parameters-7)Parameters | Name | Type | Description | | --- | --- | --- | | scheme | string | The decoding scheme to use. | #### [](#examples-7)Examples ```bloblang root.decoded = this.value.decode("hex").string() # In: {"value":"68656c6c6f20776f726c64"} # Out: {"decoded":"hello world"} ``` ```bloblang root = this.encoded.decode("ascii85") # In: {"encoded":"FD,B0+DGm>FDl80Ci\"A>F`)8BEckl6F`M&(+Cno&@/"} # Out: this is totally unstructured data ``` ### [](#decompress)decompress Decompresses a byte array using the specified decompression algorithm. Returns decompressed data as bytes. Use with data that was previously compressed using the corresponding algorithm. #### [](#parameters-8)Parameters | Name | Type | Description | | --- | --- | --- | | algorithm | string | The decompression algorithm: gzip, pgzip (parallel gzip), zlib, bzip2, flate, snappy, lz4, or zstd. | #### [](#examples-8)Examples Decompress base64-encoded compressed data: ```bloblang root = this.compressed.decode("base64").decompress("gzip") # In: {"compressed":"H4sIAN12MWkAA8tIzcnJVyjPL8pJUfBUyMkvS1UoLkhMTgUAQpDxbxgAAAA="} # Out: hello world I love space ``` Convert decompressed bytes to string for JSON output: ```bloblang root.message = this.compressed.decode("base64").decompress("gzip").string() # In: {"compressed":"H4sIAN12MWkAA8tIzcnJVyjPL8pJUfBUyMkvS1UoLkhMTgUAQpDxbxgAAAA="} # Out: {"message":"hello world I love space"} ``` ### [](#decrypt_aes)decrypt\_aes Decrypts an AES-encrypted string or byte array. #### [](#parameters-9)Parameters | Name | Type | Description | | --- | --- | --- | | scheme | string | The scheme to use for decryption, one of ctr, gcm, ofb, cbc. | | key | string | A key to decrypt with. | | iv | string | An initialization vector / nonce. | #### [](#examples-9)Examples ```bloblang let key = "2b7e151628aed2a6abf7158809cf4f3c".decode("hex") let vector = "f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff".decode("hex") root.decrypted = this.value.decode("hex").decrypt_aes("ctr", $key, $vector).string() # In: {"value":"84e9b31ff7400bdf80be7254"} # Out: {"decrypted":"hello world!"} ``` ### [](#encode)encode Encodes a string or byte array according to a chosen scheme. #### [](#parameters-10)Parameters | Name | Type | Description | | --- | --- | --- | | scheme | string | The encoding scheme to use. | #### [](#examples-10)Examples ```bloblang root.encoded = this.value.encode("hex") # In: {"value":"hello world"} # Out: {"encoded":"68656c6c6f20776f726c64"} ``` ```bloblang root.encoded = content().encode("ascii85") # In: this is totally unstructured data # Out: {"encoded":"FD,B0+DGm>FDl80Ci\"A>F`)8BEckl6F`M&(+Cno&@/"} ``` ### [](#encrypt_aes)encrypt\_aes Encrypts a string or byte array using AES encryption. #### [](#parameters-11)Parameters | Name | Type | Description | | --- | --- | --- | | scheme | string | The scheme to use for encryption, one of ctr, gcm, ofb, cbc. | | key | string | A key to encrypt with. | | iv | string | An initialization vector / nonce. | #### [](#examples-11)Examples ```bloblang let key = "2b7e151628aed2a6abf7158809cf4f3c".decode("hex") let vector = "f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff".decode("hex") root.encrypted = this.value.encrypt_aes("ctr", $key, $vector).encode("hex") # In: {"value":"hello world!"} # Out: {"encrypted":"84e9b31ff7400bdf80be7254"} ``` ### [](#hash)hash Hashes a string or byte array using a specified algorithm. #### [](#parameters-12)Parameters | Name | Type | Description | | --- | --- | --- | | algorithm | string | The hashing algorithm to use. | | key (optional) | string | An optional key to use. | | polynomial | string | An optional polynomial key to use when selecting the crc32 algorithm, otherwise ignored. Options are IEEE (default), Castagnoli and Koopman | #### [](#examples-12)Examples ```bloblang root.h1 = this.value.hash("sha1").encode("hex") root.h2 = this.value.hash("hmac_sha1","static-key").encode("hex") # In: {"value":"hello world"} # Out: {"h1":"2aae6c35c94fcfb415dbe95f408b9ce91ee846ed","h2":"d87e5f068fa08fe90bb95bc7c8344cb809179d76"} ``` The `crc32` algorithm supports options for the polynomial: ```bloblang root.h1 = this.value.hash(algorithm: "crc32", polynomial: "Castagnoli").encode("hex") root.h2 = this.value.hash(algorithm: "crc32", polynomial: "Koopman").encode("hex") # In: {"value":"hello world"} # Out: {"h1":"c99465aa","h2":"df373d3c"} ``` ### [](#uuid_v5)uuid\_v5 Generates a version 5 UUID from a namespace and name. #### [](#parameters-13)Parameters | Name | Type | Description | | --- | --- | --- | | ns (optional) | string | An optional namespace name or UUID. It supports the dns, url, oid and x500 predefined namespaces and any valid RFC-9562 UUID. If empty, the nil UUID will be used. | #### [](#examples-13)Examples ```bloblang root.id = "example".uuid_v5() ``` ```bloblang root.id = "example".uuid_v5("x500") ``` ```bloblang root.id = "example".uuid_v5("77f836b7-9f61-46c0-851e-9b6ca3535e69") ``` ## [](#geoip)GeoIP ### [](#geoip_anonymous_ip)geoip\_anonymous\_ip Looks up an IP address against a [MaxMind database file](https://www.maxmind.com/en/home) and, if found, returns an object describing the anonymous IP associated with it. #### [](#parameters-14)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A path to an mmdb (maxmind) file. | ### [](#geoip_asn)geoip\_asn Looks up an IP address against a [MaxMind database file](https://www.maxmind.com/en/home) and, if found, returns an object describing the ASN associated with it. #### [](#parameters-15)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A path to an mmdb (maxmind) file. | ### [](#geoip_city)geoip\_city Looks up an IP address against a [MaxMind database file](https://www.maxmind.com/en/home) and, if found, returns an object describing the city associated with it. #### [](#parameters-16)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A path to an mmdb (maxmind) file. | ### [](#geoip_connection_type)geoip\_connection\_type Looks up an IP address against a [MaxMind database file](https://www.maxmind.com/en/home) and, if found, returns an object describing the connection type associated with it. #### [](#parameters-17)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A path to an mmdb (maxmind) file. | ### [](#geoip_country)geoip\_country Looks up an IP address against a [MaxMind database file](https://www.maxmind.com/en/home) and, if found, returns an object describing the country associated with it. #### [](#parameters-18)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A path to an mmdb (maxmind) file. | ### [](#geoip_domain)geoip\_domain Looks up an IP address against a [MaxMind database file](https://www.maxmind.com/en/home) and, if found, returns an object describing the domain associated with it. #### [](#parameters-19)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A path to an mmdb (maxmind) file. | ### [](#geoip_enterprise)geoip\_enterprise Looks up an IP address against a [MaxMind database file](https://www.maxmind.com/en/home) and, if found, returns an object describing the enterprise associated with it. #### [](#parameters-20)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A path to an mmdb (maxmind) file. | ### [](#geoip_isp)geoip\_isp Looks up an IP address against a [MaxMind database file](https://www.maxmind.com/en/home) and, if found, returns an object describing the ISP associated with it. #### [](#parameters-21)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A path to an mmdb (maxmind) file. | ## [](#json-web-tokens)JSON web tokens ### [](#parse_jwt_es256)parse\_jwt\_es256 Parses a claims object from a JWT string encoded with ES256. This method does not validate JWT claims. #### [](#parameters-22)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The ES256 secret that was used for signing the token. | #### [](#examples-14)Examples ```bloblang root.claims = this.signed.parse_jwt_es256("""-----BEGIN PUBLIC KEY----- MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEGtLqIBePHmIhQcf0JLgc+F/4W/oI dp0Gta53G35VerNDgUUXmp78J2kfh4qLdh0XtmOMI587tCaqjvDAXfs//w== -----END PUBLIC KEY-----""") # In: {"signed":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.GIRajP9JJbpTlqSCdNEz4qpQkRvzX4Q51YnTwVyxLDM9tKjR_a8ggHWn9CWj7KG0x8J56OWtmUxn112SRTZVhQ"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} ``` ### [](#parse_jwt_es384)parse\_jwt\_es384 Parses a claims object from a JWT string encoded with ES384. This method does not validate JWT claims. #### [](#parameters-23)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The ES384 secret that was used for signing the token. | #### [](#examples-15)Examples ```bloblang root.claims = this.signed.parse_jwt_es384("""-----BEGIN PUBLIC KEY----- MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAERoz74/B6SwmLhs8X7CWhnrWyRrB13AuU 8OYeqy0qHRu9JWNw8NIavqpTmu6XPT4xcFanYjq8FbeuM11eq06C52mNmS4LLwzA 2imlFEgn85bvJoC3bnkuq4mQjwt9VxdH -----END PUBLIC KEY-----""") # In: {"signed":"eyJhbGciOiJFUzM4NCIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.H2HBSlrvQBaov2tdreGonbBexxtQB-xzaPL4-tNQZ6TVh7VH8VBcSwcWHYa1lBAHmdsKOFcB2Wk0SB7QWeGT3ptSgr-_EhDMaZ8bA5spgdpq5DsKfaKHrd7DbbQlmxNq"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} ``` ### [](#parse_jwt_es512)parse\_jwt\_es512 Parses a claims object from a JWT string encoded with ES512. This method does not validate JWT claims. #### [](#parameters-24)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The ES512 secret that was used for signing the token. | #### [](#examples-16)Examples ```bloblang root.claims = this.signed.parse_jwt_es512("""-----BEGIN PUBLIC KEY----- MIGbMBAGByqGSM49AgEGBSuBBAAjA4GGAAQAkHLdts9P56fFkyhpYQ31M/Stwt3w vpaxhlfudxnXgTO1IP4RQRgryRxZ19EUzhvWDcG3GQIckoNMY5PelsnCGnIBT2Xh 9NQkjWF5K6xS4upFsbGSAwQ+GIyyk5IPJ2LHgOyMSCVh5gRZXV3CZLzXujx/umC9 UeYyTt05zRRWuD+p5bY= -----END PUBLIC KEY-----""") # In: {"signed":"eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.ACrpLuU7TKpAnncDCpN9m85nkL55MJ45NFOBl6-nEXmNT1eIxWjiP4pwWVbFH9et_BgN14119jbL_KqEJInPYc9nAXC6dDLq0aBU-dalvNl4-O5YWpP43-Y-TBGAsWnbMTrchILJ4-AEiICe73Ck5yWPleKg9c3LtkEFWfGs7BoPRguZ"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} ``` ### [](#parse_jwt_hs256)parse\_jwt\_hs256 Parses a claims object from a JWT string encoded with HS256. This method does not validate JWT claims. #### [](#parameters-25)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The HS256 secret that was used for signing the token. | #### [](#examples-17)Examples ```bloblang root.claims = this.signed.parse_jwt_hs256("""dont-tell-anyone""") # In: {"signed":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.YwXOM8v3gHVWcQRRRQc_zDlhmLnM62fwhFYGpiA0J1A"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} ``` ### [](#parse_jwt_hs384)parse\_jwt\_hs384 Parses a claims object from a JWT string encoded with HS384. This method does not validate JWT claims. #### [](#parameters-26)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The HS384 secret that was used for signing the token. | #### [](#examples-18)Examples ```bloblang root.claims = this.signed.parse_jwt_hs384("""dont-tell-anyone""") # In: {"signed":"eyJhbGciOiJIUzM4NCIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.2Y8rf_ijwN4t8hOGGViON_GrirLkCQVbCOuax6EoZ3nluX0tCGezcJxbctlIfsQ2"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} ``` ### [](#parse_jwt_hs512)parse\_jwt\_hs512 Parses a claims object from a JWT string encoded with HS512. This method does not validate JWT claims. #### [](#parameters-27)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The HS512 secret that was used for signing the token. | #### [](#examples-19)Examples ```bloblang root.claims = this.signed.parse_jwt_hs512("""dont-tell-anyone""") # In: {"signed":"eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.utRb0urG6LGGyranZJVo5Dk0Fns1QNcSUYPN0TObQ-YzsGGB8jrxHwM5NAJccjJZzKectEUqmmKCaETZvuX4Fg"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} ``` ### [](#parse_jwt_rs256)parse\_jwt\_rs256 Parses a claims object from a JWT string encoded with RS256. This method does not validate JWT claims. #### [](#parameters-28)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The RS256 secret that was used for signing the token. | #### [](#examples-20)Examples ```bloblang root.claims = this.signed.parse_jwt_rs256("""-----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAs/ibN8r68pLMR6gRzg4S 8v8l6Q7yi8qURjkEbcNeM1rkokC7xh0I4JVTwxYSVv/JIW8qJdyspl5NIfuAVi32 WfKvSAs+NIs+DMsNPYw3yuQals4AX8hith1YDvYpr8SD44jxhz/DR9lYKZFGhXGB +7NqQ7vpTWp3BceLYocazWJgusZt7CgecIq57ycM5hjM93BvlrUJ8nQ1a46wfL/8 Cy4P0et70hzZrsjjN41KFhKY0iUwlyU41yEiDHvHDDsTMBxAZosWjSREGfJL6Mfp XOInTHs/Gg6DZMkbxjQu6L06EdJ+Q/NwglJdAXM7Zo9rNELqRig6DdvG5JesdMsO +QIDAQAB -----END PUBLIC KEY-----""") # In: {"signed":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.b0lH3jEupZZ4zoaly4Y_GCvu94HH6UKdKY96zfGNsIkPZpQLHIkZ7jMWlLlNOAd8qXlsBGP_i8H2qCKI4zlWJBGyPZgxXDzNRPVrTDfFpn4t4nBcA1WK2-ntXP3ehQxsaHcQU8Z_nsogId7Pme5iJRnoHWEnWtbwz5DLSXL3ZZNnRdrHM9MdI7QSDz9mojKDCaMpGN9sG7Xl-tGdBp1XzXuUOzG8S03mtZ1IgVR1uiBL2N6oohHIAunk8DIAmNWI-zgycTgzUGU7mvPkKH43qO8Ua1-13tCUBKKa8VxcotZ67Mxm1QAvBGoDnTKwWMwghLzs6d6WViXQg6eWlJcpBA"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} ``` ### [](#parse_jwt_rs384)parse\_jwt\_rs384 Parses a claims object from a JWT string encoded with RS384. This method does not validate JWT claims. #### [](#parameters-29)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The RS384 secret that was used for signing the token. | #### [](#examples-21)Examples ```bloblang root.claims = this.signed.parse_jwt_rs384("""-----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAs/ibN8r68pLMR6gRzg4S 8v8l6Q7yi8qURjkEbcNeM1rkokC7xh0I4JVTwxYSVv/JIW8qJdyspl5NIfuAVi32 WfKvSAs+NIs+DMsNPYw3yuQals4AX8hith1YDvYpr8SD44jxhz/DR9lYKZFGhXGB +7NqQ7vpTWp3BceLYocazWJgusZt7CgecIq57ycM5hjM93BvlrUJ8nQ1a46wfL/8 Cy4P0et70hzZrsjjN41KFhKY0iUwlyU41yEiDHvHDDsTMBxAZosWjSREGfJL6Mfp XOInTHs/Gg6DZMkbxjQu6L06EdJ+Q/NwglJdAXM7Zo9rNELqRig6DdvG5JesdMsO +QIDAQAB -----END PUBLIC KEY-----""") # In: {"signed":"eyJhbGciOiJSUzM4NCIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.orcXYBcjVE5DU7mvq4KKWFfNdXR4nEY_xupzWoETRpYmQZIozlZnM_nHxEk2dySvpXlAzVm7kgOPK2RFtGlOVaNRIa3x-pMMr-bhZTno4L8Hl4sYxOks3bWtjK7wql4uqUbqThSJB12psAXw2-S-I_FMngOPGIn4jDT9b802ottJSvTpXcy0-eKTjrV2PSkRRu-EYJh0CJZW55MNhqlt6kCGhAXfbhNazN3ASX-dmpd_JixyBKphrngr_zRA-FCn_Xf3QQDA-5INopb4Yp5QiJ7UxVqQEKI80X_JvJqz9WE1qiAw8pq5-xTen1t7zTP-HT1NbbD3kltcNa3G8acmNg"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} ``` ### [](#parse_jwt_rs512)parse\_jwt\_rs512 Parses a claims object from a JWT string encoded with RS512. This method does not validate JWT claims. #### [](#parameters-30)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The RS512 secret that was used for signing the token. | #### [](#examples-22)Examples ```bloblang root.claims = this.signed.parse_jwt_rs512("""-----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAs/ibN8r68pLMR6gRzg4S 8v8l6Q7yi8qURjkEbcNeM1rkokC7xh0I4JVTwxYSVv/JIW8qJdyspl5NIfuAVi32 WfKvSAs+NIs+DMsNPYw3yuQals4AX8hith1YDvYpr8SD44jxhz/DR9lYKZFGhXGB +7NqQ7vpTWp3BceLYocazWJgusZt7CgecIq57ycM5hjM93BvlrUJ8nQ1a46wfL/8 Cy4P0et70hzZrsjjN41KFhKY0iUwlyU41yEiDHvHDDsTMBxAZosWjSREGfJL6Mfp XOInTHs/Gg6DZMkbxjQu6L06EdJ+Q/NwglJdAXM7Zo9rNELqRig6DdvG5JesdMsO +QIDAQAB -----END PUBLIC KEY-----""") # In: {"signed":"eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.rsMp_X5HMrUqKnZJIxo27aAoscovRA6SSQYR9rq7pifIj0YHXxMyNyOBDGnvVALHKTi25VUGHpfNUW0VVMmae0A4t_ObNU6hVZHguWvetKZZq4FZpW1lgWHCMqgPGwT5_uOqwYCH6r8tJuZT3pqXeL0CY4putb1AN2w6CVp620nh3l8d3XWb4jaifycd_4CEVCqHuWDmohfug4VhmoVKlIXZkYoAQowgHlozATDssBSWdYtv107Wd2AzEoiXPu6e3pflsuXULlyqQnS4ELEKPYThFLafh1NqvZDPddqozcPZ-iODBW-xf3A4DYDdivnMYLrh73AZOGHexxu8ay6nDA"} # Out: {"claims":{"iat":1516239022,"mood":"Disdainful","sub":"1234567890"}} ``` ### [](#sign_jwt_es256)sign\_jwt\_es256 Hash and sign an object representing JSON Web Token (JWT) claims using ES256. #### [](#parameters-31)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The secret to use for signing the token. | | headers (optional) | unknown | Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. | #### [](#examples-23)Examples ```bloblang root.signed = this.claims.sign_jwt_es256("""-----BEGIN EC PRIVATE KEY----- ... signature data ... -----END EC PRIVATE KEY-----""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.-8LrOdkEiv_44ADWW08lpbq41ZmHCel58NMORPq1q4Dyw0zFhqDVLrRoSvCvuyyvgXAFb9IHfR-9MlJ_2ShA9A"} ``` ```bloblang root.signed = this.claims.sign_jwt_es256(signing_secret: """-----BEGIN EC PRIVATE KEY----- ... signature data ... -----END EC PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ### [](#sign_jwt_es384)sign\_jwt\_es384 Hash and sign an object representing JSON Web Token (JWT) claims using ES384. #### [](#parameters-32)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The secret to use for signing the token. | | headers (optional) | unknown | Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. | #### [](#examples-24)Examples ```bloblang root.signed = this.claims.sign_jwt_es384("""-----BEGIN EC PRIVATE KEY----- ... signature data ... -----END EC PRIVATE KEY-----""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":"eyJhbGciOiJFUzM4NCIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyMTIzIn0.8FmTKH08dl7dyxrNu0rmvhegiIBCy-O9cddGco2e9lpZtgv5mS5qHgPkgBC5eRw1d7SRJsHwHZeehzdqT5Ba7aZJIhz9ds0sn37YQ60L7jT0j2gxCzccrt4kECHnUnLw"} ``` ```bloblang root.signed = this.claims.sign_jwt_es384(signing_secret: """-----BEGIN EC PRIVATE KEY----- ... signature data ... -----END EC PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ### [](#sign_jwt_es512)sign\_jwt\_es512 Hash and sign an object representing JSON Web Token (JWT) claims using ES512. #### [](#parameters-33)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The secret to use for signing the token. | | headers (optional) | unknown | Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. | #### [](#examples-25)Examples ```bloblang root.signed = this.claims.sign_jwt_es512("""-----BEGIN EC PRIVATE KEY----- ... signature data ... -----END EC PRIVATE KEY-----""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":"eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyMTIzIn0.AQbEWymoRZxDJEJtKSFFG2k2VbDCTYSuBwAZyMqexCspr3If8aERTVGif8HXG3S7TzMBCCzxkcKr3eIU441l3DlpAMNfQbkcOlBqMvNBn-CX481WyKf3K5rFHQ-6wRonz05aIsWAxCDvAozI_9J0OWllxdQ2MBAuTPbPJ38OqXsYkCQs"} ``` ```bloblang root.signed = this.claims.sign_jwt_es512(signing_secret: """-----BEGIN EC PRIVATE KEY----- ... signature data ... -----END EC PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ### [](#sign_jwt_hs256)sign\_jwt\_hs256 Hash and sign an object representing JSON Web Token (JWT) claims using HS256. #### [](#parameters-34)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The secret to use for signing the token. | | headers (optional) | unknown | Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. | #### [](#examples-26)Examples ```bloblang root.signed = this.claims.sign_jwt_hs256("""dont-tell-anyone""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyMTIzIn0.hUl-nngPMY_3h9vveWJUPsCcO5PeL6k9hWLnMYeFbFQ"} ``` ```bloblang root.signed = this.claims.sign_jwt_hs256(signing_secret: """dont-tell-anyone""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ### [](#sign_jwt_hs384)sign\_jwt\_hs384 Hash and sign an object representing JSON Web Token (JWT) claims using HS384. #### [](#parameters-35)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The secret to use for signing the token. | | headers (optional) | unknown | Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. | #### [](#examples-27)Examples ```bloblang root.signed = this.claims.sign_jwt_hs384("""dont-tell-anyone""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":"eyJhbGciOiJIUzM4NCIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyMTIzIn0.zGYLr83aToon1efUNq-hw7XgT20lPvZb8sYei8x6S6mpHwb433SJdXJXx0Oio8AZ"} ``` ```bloblang root.signed = this.claims.sign_jwt_hs384(signing_secret: """dont-tell-anyone""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ### [](#sign_jwt_hs512)sign\_jwt\_hs512 Hash and sign an object representing JSON Web Token (JWT) claims using HS512. #### [](#parameters-36)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The secret to use for signing the token. | | headers (optional) | unknown | Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. | #### [](#examples-28)Examples ```bloblang root.signed = this.claims.sign_jwt_hs512("""dont-tell-anyone""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":"eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyMTIzIn0.zBNR9o_6EDwXXKkpKLNJhG26j8Dc-mV-YahBwmEdCrmiWt5les8I9rgmNlWIowpq6Yxs4kLNAdFhqoRz3NXT3w"} ``` ```bloblang root.signed = this.claims.sign_jwt_hs512(signing_secret: """dont-tell-anyone""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ### [](#sign_jwt_rs256)sign\_jwt\_rs256 Hash and sign an object representing JSON Web Token (JWT) claims using RS256. #### [](#parameters-37)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The secret to use for signing the token. | | headers (optional) | unknown | Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. | #### [](#examples-29)Examples ```bloblang root.signed = this.claims.sign_jwt_rs256("""-----BEGIN RSA PRIVATE KEY----- ... signature data ... -----END RSA PRIVATE KEY-----""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.b0lH3jEupZZ4zoaly4Y_GCvu94HH6UKdKY96zfGNsIkPZpQLHIkZ7jMWlLlNOAd8qXlsBGP_i8H2qCKI4zlWJBGyPZgxXDzNRPVrTDfFpn4t4nBcA1WK2-ntXP3ehQxsaHcQU8Z_nsogId7Pme5iJRnoHWEnWtbwz5DLSXL3ZZNnRdrHM9MdI7QSDz9mojKDCaMpGN9sG7Xl-tGdBp1XzXuUOzG8S03mtZ1IgVR1uiBL2N6oohHIAunk8DIAmNWI-zgycTgzUGU7mvPkKH43qO8Ua1-13tCUBKKa8VxcotZ67Mxm1QAvBGoDnTKwWMwghLzs6d6WViXQg6eWlJcpBA"} ``` ```bloblang root.signed = this.claims.sign_jwt_rs256(signing_secret: """-----BEGIN RSA PRIVATE KEY----- ... signature data ... -----END RSA PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ### [](#sign_jwt_rs384)sign\_jwt\_rs384 Hash and sign an object representing JSON Web Token (JWT) claims using RS384. #### [](#parameters-38)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The secret to use for signing the token. | | headers (optional) | unknown | Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. | #### [](#examples-30)Examples ```bloblang root.signed = this.claims.sign_jwt_rs384("""-----BEGIN RSA PRIVATE KEY----- ... signature data ... -----END RSA PRIVATE KEY-----""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":"eyJhbGciOiJSUzM4NCIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.orcXYBcjVE5DU7mvq4KKWFfNdXR4nEY_xupzWoETRpYmQZIozlZnM_nHxEk2dySvpXlAzVm7kgOPK2RFtGlOVaNRIa3x-pMMr-bhZTno4L8Hl4sYxOks3bWtjK7wql4uqUbqThSJB12psAXw2-S-I_FMngOPGIn4jDT9b802ottJSvTpXcy0-eKTjrV2PSkRRu-EYJh0CJZW55MNhqlt6kCGhAXfbhNazN3ASX-dmpd_JixyBKphrngr_zRA-FCn_Xf3QQDA-5INopb4Yp5QiJ7UxVqQEKI80X_JvJqz9WE1qiAw8pq5-xTen1t7zTP-HT1NbbD3kltcNa3G8acmNg"} ``` ```bloblang root.signed = this.claims.sign_jwt_rs384(signing_secret: """-----BEGIN RSA PRIVATE KEY----- ... signature data ... -----END RSA PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ### [](#sign_jwt_rs512)sign\_jwt\_rs512 Hash and sign an object representing JSON Web Token (JWT) claims using RS512. #### [](#parameters-39)Parameters | Name | Type | Description | | --- | --- | --- | | signing_secret | string | The secret to use for signing the token. | | headers (optional) | unknown | Optional object of JWT header fields to include in the token. Keys "alg", "typ", "jku", "jwk", "x5u", "x5c", "x5t","x5t#S256" and "crit" will be ignored if provided. | #### [](#examples-31)Examples ```bloblang root.signed = this.claims.sign_jwt_rs512("""-----BEGIN RSA PRIVATE KEY----- ... signature data ... -----END RSA PRIVATE KEY-----""") # In: {"claims":{"sub":"user123"}} # Out: {"signed":"eyJhbGciOiJSUzUxMiIsInR5cCI6IkpXVCJ9.eyJpYXQiOjE1MTYyMzkwMjIsIm1vb2QiOiJEaXNkYWluZnVsIiwic3ViIjoiMTIzNDU2Nzg5MCJ9.rsMp_X5HMrUqKnZJIxo27aAoscovRA6SSQYR9rq7pifIj0YHXxMyNyOBDGnvVALHKTi25VUGHpfNUW0VVMmae0A4t_ObNU6hVZHguWvetKZZq4FZpW1lgWHCMqgPGwT5_uOqwYCH6r8tJuZT3pqXeL0CY4putb1AN2w6CVp620nh3l8d3XWb4jaifycd_4CEVCqHuWDmohfug4VhmoVKlIXZkYoAQowgHlozATDssBSWdYtv107Wd2AzEoiXPu6e3pflsuXULlyqQnS4ELEKPYThFLafh1NqvZDPddqozcPZ-iODBW-xf3A4DYDdivnMYLrh73AZOGHexxu8ay6nDA"} ``` ```bloblang root.signed = this.claims.sign_jwt_rs512(signing_secret: """-----BEGIN RSA PRIVATE KEY----- ... signature data ... -----END RSA PRIVATE KEY-----""", headers: {"kid": "my-key", "x": "y"}) # In: {"claims":{"sub":"user123"}} # Out: {"signed":""} ``` ## [](#number-manipulation)Number manipulation ### [](#abs)abs Returns the absolute value of an int64 or float64 number. As a special case, when an integer is provided that is the minimum value it is converted to the maximum value. #### [](#examples-32)Examples ```bloblang root.outs = this.ins.map_each(ele -> ele.abs()) # In: {"ins":[9,-18,1.23,-4.56]} # Out: {"outs":[9,18,1.23,4.56]} ``` ### [](#bitwise_and)bitwise\_and Performs a bitwise AND operation between the integer and the specified value. #### [](#parameters-40)Parameters | Name | Type | Description | | --- | --- | --- | | value | integer | The value to AND with | #### [](#examples-33)Examples ```bloblang root.new_value = this.value.bitwise_and(6) # In: {"value":12} # Out: {"new_value":4} ``` ```bloblang root.masked = this.flags.bitwise_and(15) # In: {"flags":127} # Out: {"masked":15} ``` ### [](#bitwise_or)bitwise\_or Performs a bitwise OR operation between the integer and the specified value. #### [](#parameters-41)Parameters | Name | Type | Description | | --- | --- | --- | | value | integer | The value to OR with | #### [](#examples-34)Examples ```bloblang root.new_value = this.value.bitwise_or(6) # In: {"value":12} # Out: {"new_value":14} ``` ```bloblang root.combined = this.flags.bitwise_or(8) # In: {"flags":4} # Out: {"combined":12} ``` ### [](#bitwise_xor)bitwise\_xor Performs a bitwise XOR (exclusive OR) operation between the integer and the specified value. #### [](#parameters-42)Parameters | Name | Type | Description | | --- | --- | --- | | value | integer | The value to XOR with | #### [](#examples-35)Examples ```bloblang root.new_value = this.value.bitwise_xor(6) # In: {"value":12} # Out: {"new_value":10} ``` ```bloblang root.toggled = this.flags.bitwise_xor(5) # In: {"flags":3} # Out: {"toggled":6} ``` ### [](#ceil)ceil Rounds a number up to the nearest integer. Returns an integer if the result fits in 64-bit, otherwise returns a float. #### [](#examples-36)Examples ```bloblang root.new_value = this.value.ceil() # In: {"value":5.3} # Out: {"new_value":6} # In: {"value":-5.9} # Out: {"new_value":-5} ``` ```bloblang root.result = this.price.ceil() # In: {"price":19.99} # Out: {"result":20} ``` ### [](#cos)cos Calculates the cosine of a given angle specified in radians. #### [](#examples-37)Examples ```bloblang root.new_value = (this.value * (pi() / 180)).cos() # In: {"value":45} # Out: {"new_value":0.7071067811865476} # In: {"value":0} # Out: {"new_value":1} # In: {"value":180} # Out: {"new_value":-1} ``` ### [](#float32)float32 Converts a numerical type into a 32-bit floating point number, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 32-bit floating point number. Please refer to the [`strconv.ParseFloat` documentation](https://pkg.go.dev/strconv#ParseFloat) for details regarding the supported formats. #### [](#examples-38)Examples ```bloblang root.out = this.in.float32() # In: {"in":"6.674282313423543523453425345e-11"} # Out: {"out":6.674283e-11} ``` ### [](#float64)float64 Converts a numerical type into a 64-bit floating point number, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 64-bit floating point number. Please refer to the [`strconv.ParseFloat` documentation](https://pkg.go.dev/strconv#ParseFloat) for details regarding the supported formats. #### [](#examples-39)Examples ```bloblang root.out = this.in.float64() # In: {"in":"6.674282313423543523453425345e-11"} # Out: {"out":6.674282313423544e-11} ``` ### [](#floor)floor Rounds a number down to the nearest integer. Returns an integer if the result fits in 64-bit, otherwise returns a float. #### [](#examples-40)Examples ```bloblang root.new_value = this.value.floor() # In: {"value":5.7} # Out: {"new_value":5} # In: {"value":-3.2} # Out: {"new_value":-4} ``` ```bloblang root.whole_seconds = this.duration_seconds.floor() # In: {"duration_seconds":12.345} # Out: {"whole_seconds":12} ``` ### [](#int16)int16 Converts a numerical type into a 16-bit signed integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 16-bit signed integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use [`.round()`](#round) on the value. Please refer to the [`strconv.ParseInt` documentation](https://pkg.go.dev/strconv#ParseInt) for details regarding the supported formats. #### [](#examples-41)Examples ```bloblang root.a = this.a.int16() root.b = this.b.round().int16() root.c = this.c.int16() root.d = this.d.int16().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":-12} ``` ```bloblang root = this.int16() # In: "0xDE" # Out: 222 ``` ### [](#int32)int32 Converts a numerical type into a 32-bit signed integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 32-bit signed integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use [`.round()`](#round) on the value. Please refer to the [`strconv.ParseInt` documentation](https://pkg.go.dev/strconv#ParseInt) for details regarding the supported formats. #### [](#examples-42)Examples ```bloblang root.a = this.a.int32() root.b = this.b.round().int32() root.c = this.c.int32() root.d = this.d.int32().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":-12} ``` ```bloblang root = this.int32() # In: "0xDEAD" # Out: 57005 ``` ### [](#int64)int64 Converts a numerical type into a 64-bit signed integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 64-bit signed integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use [`.round()`](#round) on the value. Please refer to the [`strconv.ParseInt` documentation](https://pkg.go.dev/strconv#ParseInt) for details regarding the supported formats. #### [](#examples-43)Examples ```bloblang root.a = this.a.int64() root.b = this.b.round().int64() root.c = this.c.int64() root.d = this.d.int64().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":-12} ``` ```bloblang root = this.int64() # In: "0xDEADBEEF" # Out: 3735928559 ``` ### [](#int8)int8 Converts a numerical type into a 8-bit signed integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 8-bit signed integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use [`.round()`](#round) on the value. Please refer to the [`strconv.ParseInt` documentation](https://pkg.go.dev/strconv#ParseInt) for details regarding the supported formats. #### [](#examples-44)Examples ```bloblang root.a = this.a.int8() root.b = this.b.round().int8() root.c = this.c.int8() root.d = this.d.int8().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":-12} ``` ```bloblang root = this.int8() # In: "0xD" # Out: 13 ``` ### [](#log)log Calculates the natural logarithm (base e) of a number. #### [](#examples-45)Examples ```bloblang root.new_value = this.value.log().round() # In: {"value":1} # Out: {"new_value":0} # In: {"value":2.7183} # Out: {"new_value":1} ``` ```bloblang root.ln_result = this.number.log() # In: {"number":10} # Out: {"ln_result":2.302585092994046} ``` ### [](#log10)log10 Calculates the base-10 logarithm of a number. #### [](#examples-46)Examples ```bloblang root.new_value = this.value.log10() # In: {"value":100} # Out: {"new_value":2} # In: {"value":1000} # Out: {"new_value":3} ``` ```bloblang root.log_value = this.magnitude.log10() # In: {"magnitude":10000} # Out: {"log_value":4} ``` ### [](#max)max Returns the largest number from an array. All elements must be numbers and the array cannot be empty. #### [](#examples-47)Examples ```bloblang root.biggest = this.values.max() # In: {"values":[0,3,2.5,7,5]} # Out: {"biggest":7} ``` ```bloblang root.highest_temp = this.temperatures.max() # In: {"temperatures":[20.5,22.1,19.8,23.4]} # Out: {"highest_temp":23.4} ``` ### [](#min)min Returns the smallest number from an array. All elements must be numbers and the array cannot be empty. #### [](#examples-48)Examples ```bloblang root.smallest = this.values.min() # In: {"values":[0,3,-2.5,7,5]} # Out: {"smallest":-2.5} ``` ```bloblang root.lowest_temp = this.temperatures.min() # In: {"temperatures":[20.5,22.1,19.8,23.4]} # Out: {"lowest_temp":19.8} ``` ### [](#pow)pow Returns the number raised to the specified exponent. #### [](#parameters-43)Parameters | Name | Type | Description | | --- | --- | --- | | exponent | float | The exponent you want to raise to the power of. | #### [](#examples-49)Examples ```bloblang root.new_value = this.value * 10.pow(-2) # In: {"value":2} # Out: {"new_value":0.02} ``` ```bloblang root.new_value = this.value.pow(-2) # In: {"value":2} # Out: {"new_value":0.25} ``` ### [](#round)round Rounds a number to the nearest integer. Values at .5 round away from zero. Returns an integer if the result fits in 64-bit, otherwise returns a float. #### [](#examples-50)Examples ```bloblang root.new_value = this.value.round() # In: {"value":5.3} # Out: {"new_value":5} # In: {"value":5.9} # Out: {"new_value":6} ``` ```bloblang root.rounded = this.score.round() # In: {"score":87.5} # Out: {"rounded":88} ``` ### [](#sin)sin Calculates the sine of a given angle specified in radians. #### [](#examples-51)Examples ```bloblang root.new_value = (this.value * (pi() / 180)).sin() # In: {"value":45} # Out: {"new_value":0.7071067811865475} # In: {"value":0} # Out: {"new_value":0} # In: {"value":90} # Out: {"new_value":1} ``` ### [](#tan)tan Calculates the tangent of a given angle specified in radians. #### [](#examples-52)Examples ```bloblang root.new_value = "%f".format((this.value * (pi() / 180)).tan()) # In: {"value":0} # Out: {"new_value":"0.000000"} # In: {"value":45} # Out: {"new_value":"1.000000"} # In: {"value":180} # Out: {"new_value":"-0.000000"} ``` ### [](#uint16)uint16 Converts a numerical type into a 16-bit unsigned integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 16-bit unsigned integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use [`.round()`](#round) on the value. Please refer to the [`strconv.ParseInt` documentation](https://pkg.go.dev/strconv#ParseInt) for details regarding the supported formats. #### [](#examples-53)Examples ```bloblang root.a = this.a.uint16() root.b = this.b.round().uint16() root.c = this.c.uint16() root.d = this.d.uint16().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":0} ``` ```bloblang root = this.uint16() # In: "0xDE" # Out: 222 ``` ### [](#uint32)uint32 Converts a numerical type into a 32-bit unsigned integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 32-bit unsigned integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use [`.round()`](#round) on the value. Please refer to the [`strconv.ParseInt` documentation](https://pkg.go.dev/strconv#ParseInt) for details regarding the supported formats. #### [](#examples-54)Examples ```bloblang root.a = this.a.uint32() root.b = this.b.round().uint32() root.c = this.c.uint32() root.d = this.d.uint32().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":0} ``` ```bloblang root = this.uint32() # In: "0xDEAD" # Out: 57005 ``` ### [](#uint64)uint64 Converts a numerical type into a 64-bit unsigned integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 64-bit unsigned integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use [`.round()`](#round) on the value. Please refer to the [`strconv.ParseInt` documentation](https://pkg.go.dev/strconv#ParseInt) for details regarding the supported formats. #### [](#examples-55)Examples ```bloblang root.a = this.a.uint64() root.b = this.b.round().uint64() root.c = this.c.uint64() root.d = this.d.uint64().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":0} ``` ```bloblang root = this.uint64() # In: "0xDEADBEEF" # Out: 3735928559 ``` ### [](#uint8)uint8 Converts a numerical type into a 8-bit unsigned integer, this is for advanced use cases where a specific data type is needed for a given component (such as the ClickHouse SQL driver). If the value is a string then an attempt will be made to parse it as a 8-bit unsigned integer. If the target value exceeds the capacity of an integer or contains decimal values then this method will throw an error. In order to convert a floating point number containing decimals first use [`.round()`](#round) on the value. Please refer to the [`strconv.ParseInt` documentation](https://pkg.go.dev/strconv#ParseInt) for details regarding the supported formats. #### [](#examples-56)Examples ```bloblang root.a = this.a.uint8() root.b = this.b.round().uint8() root.c = this.c.uint8() root.d = this.d.uint8().catch(0) # In: {"a":12,"b":12.34,"c":"12","d":-12} # Out: {"a":12,"b":12,"c":12,"d":0} ``` ```bloblang root = this.uint8() # In: "0xD" # Out: 13 ``` ## [](#object-array-manipulation)Object & array manipulation ### [](#all)all Tests whether all elements in an array satisfy a condition. Returns true only if the query evaluates to true for every element. Returns false for empty arrays. #### [](#parameters-44)Parameters | Name | Type | Description | | --- | --- | --- | | test | query expression | A test query to apply to each element. | #### [](#examples-57)Examples ```bloblang root.all_over_21 = this.patrons.all(patron -> patron.age >= 21) # In: {"patrons":[{"id":"1","age":18},{"id":"2","age":23}]} # Out: {"all_over_21":false} # In: {"patrons":[{"id":"1","age":45},{"id":"2","age":23}]} # Out: {"all_over_21":true} ``` ```bloblang root.all_positive = this.values.all(v -> v > 0) # In: {"values":[1,2,3,4,5]} # Out: {"all_positive":true} # In: {"values":[1,-2,3,4,5]} # Out: {"all_positive":false} ``` ### [](#any)any Tests whether at least one element in an array satisfies a condition. Returns true if the query evaluates to true for any element. Returns false for empty arrays. #### [](#parameters-45)Parameters | Name | Type | Description | | --- | --- | --- | | test | query expression | A test query to apply to each element. | #### [](#examples-58)Examples ```bloblang root.any_over_21 = this.patrons.any(patron -> patron.age >= 21) # In: {"patrons":[{"id":"1","age":18},{"id":"2","age":23}]} # Out: {"any_over_21":true} # In: {"patrons":[{"id":"1","age":10},{"id":"2","age":12}]} # Out: {"any_over_21":false} ``` ```bloblang root.has_errors = this.results.any(r -> r.status == "error") # In: {"results":[{"status":"ok"},{"status":"error"},{"status":"ok"}]} # Out: {"has_errors":true} # In: {"results":[{"status":"ok"},{"status":"ok"}]} # Out: {"has_errors":false} ``` ### [](#append)append Adds one or more elements to the end of an array and returns the new array. The original array is not modified. #### [](#examples-59)Examples ```bloblang root.foo = this.foo.append("and", "this") # In: {"foo":["bar","baz"]} # Out: {"foo":["bar","baz","and","this"]} ``` ```bloblang root.combined = this.items.append(this.new_item) # In: {"items":["apple","banana"],"new_item":"orange"} # Out: {"combined":["apple","banana","orange"]} ``` ### [](#assign)assign Merges two objects or arrays with override behavior. For objects, source values replace destination values on key conflicts. Arrays are concatenated. To preserve both values on conflict, use the merge method instead. #### [](#parameters-46)Parameters | Name | Type | Description | | --- | --- | --- | | with | unknown | A value to merge the target value with. | #### [](#examples-60)Examples ```bloblang root = this.foo.assign(this.bar) # In: {"foo":{"first_name":"fooer","likes":"bars"},"bar":{"second_name":"barer","likes":"foos"}} # Out: {"first_name":"fooer","likes":"foos","second_name":"barer"} ``` Override defaults with user settings: ```bloblang root.config = this.defaults.assign(this.user_settings) # In: {"defaults":{"timeout":30,"retries":3},"user_settings":{"timeout":60}} # Out: {"config":{"retries":3,"timeout":60}} ``` ### [](#collapse)collapse Flattens a nested structure into a flat object with dot-notation keys. #### [](#parameters-47)Parameters | Name | Type | Description | | --- | --- | --- | | include_empty | bool | Whether to include empty objects and arrays in the resulting object. | #### [](#examples-61)Examples ```bloblang root.result = this.collapse() # In: {"foo":[{"bar":"1"},{"bar":{}},{"bar":"2"},{"bar":[]}]} # Out: {"result":{"foo.0.bar":"1","foo.2.bar":"2"}} ``` Set include\_empty to true to preserve empty objects and arrays in the output: ```bloblang root.result = this.collapse(include_empty: true) # In: {"foo":[{"bar":"1"},{"bar":{}},{"bar":"2"},{"bar":[]}]} # Out: {"result":{"foo.0.bar":"1","foo.1.bar":{},"foo.2.bar":"2","foo.3.bar":[]}} ``` ### [](#concat)concat Concatenates an array value with one or more argument arrays. #### [](#examples-62)Examples ```bloblang root.foo = this.foo.concat(this.bar, this.baz) # In: {"foo":["a","b"],"bar":["c"],"baz":["d","e","f"]} # Out: {"foo":["a","b","c","d","e","f"]} ``` ### [](#contains)contains Tests if an array or object contains a value. #### [](#parameters-48)Parameters | Name | Type | Description | | --- | --- | --- | | value | unknown | A value to test against elements of the target. | #### [](#examples-63)Examples ```bloblang root.has_foo = this.thing.contains("foo") # In: {"thing":["this","foo","that"]} # Out: {"has_foo":true} # In: {"thing":["this","bar","that"]} # Out: {"has_foo":false} ``` ```bloblang root.has_bar = this.thing.contains(20) # In: {"thing":[10.3,20.0,"huh",3]} # Out: {"has_bar":true} # In: {"thing":[2,3,40,67]} # Out: {"has_bar":false} ``` ```bloblang root.has_foo = this.thing.contains("foo") # In: {"thing":"this foo that"} # Out: {"has_foo":true} # In: {"thing":"this bar that"} # Out: {"has_foo":false} ``` ### [](#diff)diff Compares the current value with another value and returns a detailed changelog describing all differences. The changelog contains operations (create, update, delete) with their paths and values, enabling you to track changes between data versions, implement audit logs, or synchronize data between systems. #### [](#parameters-49)Parameters | Name | Type | Description | | --- | --- | --- | | other | unknown | The value to compare against the current value. Can be any structured data (object or array). | #### [](#examples-64)Examples Compare two objects to track field changes: ```bloblang root.changes = this.before.diff(this.after) # In: {"before":{"name":"Alice","age":30},"after":{"name":"Alice","age":31,"city":"NYC"}} # Out: {"changes":[{"From":30,"Path":["age"],"To":31,"Type":"update"},{"From":null,"Path":["city"],"To":"NYC","Type":"create"}]} ``` Detect deletions in configuration changes: ```bloblang root.changelog = this.old_config.diff(this.new_config) # In: {"old_config":{"debug":true,"timeout":30},"new_config":{"timeout":60}} # Out: {"changelog":[{"From":true,"Path":["debug"],"To":null,"Type":"delete"},{"From":30,"Path":["timeout"],"To":60,"Type":"update"}]} ``` ### [](#enumerated)enumerated Transforms an array into an array of objects with index and value fields, making it easy to access both the position and content of each element. #### [](#examples-65)Examples ```bloblang root.foo = this.foo.enumerated() # In: {"foo":["bar","baz"]} # Out: {"foo":[{"index":0,"value":"bar"},{"index":1,"value":"baz"}]} ``` Useful for filtering by index position: ```bloblang root.first_two = this.items.enumerated().filter(item -> item.index < 2).map_each(item -> item.value) # In: {"items":["a","b","c","d"]} # Out: {"first_two":["a","b"]} ``` ### [](#exists)exists Checks whether a field exists at the specified dot path within an object. Returns true if the field is present (even if null), false otherwise. #### [](#parameters-50)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A dot path to a field. | #### [](#examples-66)Examples ```bloblang root.result = this.foo.exists("bar.baz") # In: {"foo":{"bar":{"baz":"yep, I exist"}}} # Out: {"result":true} # In: {"foo":{"bar":{}}} # Out: {"result":false} # In: {"foo":{}} # Out: {"result":false} ``` Also returns true for null values if the field exists: ```bloblang root.has_field = this.data.exists("optional_field") # In: {"data":{"optional_field":null}} # Out: {"has_field":true} # In: {"data":{}} # Out: {"has_field":false} ``` ### [](#explode)explode Expands a nested field into multiple documents. #### [](#parameters-51)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A dot path to a field to explode. | #### [](#examples-67)Examples ##### [](#on-arrays)On arrays When exploding an array, each element becomes a separate document with the array element replacing the original field: ```bloblang root = this.explode("value") # In: {"id":1,"value":["foo","bar","baz"]} # Out: [{"id":1,"value":"foo"},{"id":1,"value":"bar"},{"id":1,"value":"baz"}] ``` ##### [](#on-objects)On objects When exploding an object, the output keys match the nested object’s keys, with values being the full document where the target field is replaced by each nested value: ```bloblang root = this.explode("value") # In: {"id":1,"value":{"foo":2,"bar":[3,4],"baz":{"bev":5}}} # Out: {"bar":{"id":1,"value":[3,4]},"baz":{"id":1,"value":{"bev":5}},"foo":{"id":1,"value":2}} ``` ### [](#filter)filter Filters array or object elements based on a condition. #### [](#parameters-52)Parameters | Name | Type | Description | | --- | --- | --- | | test | query expression | A query to apply to each element, if this query resolves to any value other than a boolean true the element will be removed from the result. | #### [](#examples-68)Examples ```bloblang root.new_nums = this.nums.filter(num -> num > 10) # In: {"nums":[3,11,4,17]} # Out: {"new_nums":[11,17]} ``` ##### [](#on-objects-2)On objects When filtering objects, the query receives a context with `key` and `value` fields for each entry: ```bloblang root.new_dict = this.dict.filter(item -> item.value.contains("foo")) # In: {"dict":{"first":"hello foo","second":"world","third":"this foo is great"}} # Out: {"new_dict":{"first":"hello foo","third":"this foo is great"}} ``` ### [](#find)find Searches an array for a matching value and returns the index of the first occurrence. Returns -1 if no match is found. Numeric types are compared by value regardless of representation. #### [](#parameters-53)Parameters | Name | Type | Description | | --- | --- | --- | | value | unknown | A value to find. | #### [](#examples-69)Examples ```bloblang root.index = this.find("bar") # In: ["foo", "bar", "baz"] # Out: {"index":1} ``` ```bloblang root.index = this.things.find(this.goal) # In: {"goal":"bar","things":["foo", "bar", "baz"]} # Out: {"index":1} ``` ### [](#find_all)find\_all Searches an array for all occurrences of a value and returns an array of matching indexes. Returns an empty array if no matches are found. Numeric types are compared by value regardless of representation. #### [](#parameters-54)Parameters | Name | Type | Description | | --- | --- | --- | | value | unknown | A value to find. | #### [](#examples-70)Examples ```bloblang root.index = this.find_all("bar") # In: ["foo", "bar", "baz", "bar"] # Out: {"index":[1,3]} ``` ```bloblang root.indexes = this.things.find_all(this.goal) # In: {"goal":"bar","things":["foo", "bar", "baz", "bar", "buz"]} # Out: {"indexes":[1,3]} ``` ### [](#find_all_by)find\_all\_by Searches an array for all elements that satisfy a condition and returns an array of their indexes. Returns an empty array if no elements match. #### [](#parameters-55)Parameters | Name | Type | Description | | --- | --- | --- | | query | query expression | A query to execute for each element. | #### [](#examples-71)Examples ```bloblang root.index = this.find_all_by(v -> v != "bar") # In: ["foo", "bar", "baz"] # Out: {"index":[0,2]} ``` Find all indexes matching criteria: ```bloblang root.error_indexes = this.logs.find_all_by(log -> log.level == "error") # In: {"logs":[{"level":"info"},{"level":"error"},{"level":"warn"},{"level":"error"}]} # Out: {"error_indexes":[1,3]} ``` ### [](#find_by)find\_by Searches an array for the first element that satisfies a condition and returns its index. Returns -1 if no element matches the query. #### [](#parameters-56)Parameters | Name | Type | Description | | --- | --- | --- | | query | query expression | A query to execute for each element. | #### [](#examples-72)Examples ```bloblang root.index = this.find_by(v -> v != "bar") # In: ["foo", "bar", "baz"] # Out: {"index":0} ``` Find first object matching criteria: ```bloblang root.first_adult = this.users.find_by(u -> u.age >= 18) # In: {"users":[{"name":"Alice","age":15},{"name":"Bob","age":22},{"name":"Carol","age":19}]} # Out: {"first_adult":1} ``` ### [](#flatten)flatten Flattens an array by one level, expanding nested arrays into the parent array. Only the first level of nesting is removed. #### [](#examples-73)Examples ```bloblang root.result = this.flatten() # In: ["foo",["bar","baz"],"buz"] # Out: {"result":["foo","bar","baz","buz"]} ``` Deeper nesting requires multiple flatten calls: ```bloblang root.result = this.data.flatten() # In: {"data":["a",["b",["c","d"]],"e"]} # Out: {"result":["a","b",["c","d"],"e"]} ``` ### [](#fold)fold Reduces an array to a single value by iteratively applying a function. Also known as reduce or aggregate. The query receives an accumulator (tally) and current element (value) for each iteration. #### [](#parameters-57)Parameters | Name | Type | Description | | --- | --- | --- | | initial | unknown | The initial value to start the fold with. For example, an empty object {}, a zero count 0, or an empty string "". | | query | query expression | A query to apply for each element. The query is provided an object with two fields; tally containing the current tally, and value containing the value of the current element. The query should result in a new tally to be passed to the next element query. | #### [](#examples-74)Examples Sum numbers in an array: ```bloblang root.sum = this.foo.fold(0, item -> item.tally + item.value) # In: {"foo":[3,8,11]} # Out: {"sum":22} ``` Concatenate strings: ```bloblang root.result = this.foo.fold("", item -> "%v%v".format(item.tally, item.value)) # In: {"foo":["hello ", "world"]} # Out: {"result":"hello world"} ``` Merge an array of objects into a single object: ```bloblang root.smoothie = this.fruits.fold({}, item -> item.tally.merge(item.value)) # In: {"fruits":[{"apple":5},{"banana":3},{"orange":8}]} # Out: {"smoothie":{"apple":5,"banana":3,"orange":8}} ``` ### [](#get)get Extract a field value, identified via a [dot path](../../../configuration/field_paths/), from an object. #### [](#parameters-58)Parameters | Name | Type | Description | | --- | --- | --- | | path | string | A dot path identifying a field to obtain. | #### [](#examples-75)Examples ```bloblang root.result = this.foo.get(this.target) # In: {"foo":{"bar":"from bar","baz":"from baz"},"target":"bar"} # Out: {"result":"from bar"} # In: {"foo":{"bar":"from bar","baz":"from baz"},"target":"baz"} # Out: {"result":"from baz"} ``` ### [](#index)index Extract an element from an array by an index. The index can be negative, and if so the element will be selected from the end counting backwards starting from -1. E.g. an index of -1 returns the last element, an index of -2 returns the element before the last, and so on. #### [](#parameters-59)Parameters | Name | Type | Description | | --- | --- | --- | | index | integer | The index to obtain from an array. | #### [](#examples-76)Examples ```bloblang root.last_name = this.names.index(-1) # In: {"names":["rachel","stevens"]} # Out: {"last_name":"stevens"} ``` It is also possible to use this method on byte arrays, in which case the selected element will be returned as an integer: ```bloblang root.last_byte = this.name.bytes().index(-1) # In: {"name":"foobar bazson"} # Out: {"last_byte":110} ``` ### [](#join)join Joins an array of strings with an optional delimiter. #### [](#parameters-60)Parameters | Name | Type | Description | | --- | --- | --- | | delimiter (optional) | string | An optional delimiter to add between each string. | #### [](#examples-77)Examples ```bloblang root.joined_words = this.words.join() root.joined_numbers = this.numbers.map_each(this.string()).join(",") # In: {"words":["hello","world"],"numbers":[3,8,11]} # Out: {"joined_numbers":"3,8,11","joined_words":"helloworld"} ``` ### [](#json_path)json\_path Executes the given JSONPath expression on an object or array and returns the result. The JSONPath expression syntax can be found at [https://goessner.net/articles/JsonPath/](https://goessner.net/articles/JsonPath/). For more complex logic, you can use Gval expressions ([https://github.com/PaesslerAG/gval](https://github.com/PaesslerAG/gval)). #### [](#parameters-61)Parameters | Name | Type | Description | | --- | --- | --- | | expression | string | The JSONPath expression to execute. | #### [](#examples-78)Examples ```bloblang root.all_names = this.json_path("$..name") # In: {"name":"alice","foo":{"name":"bob"}} # Out: {"all_names":["alice","bob"]} # In: {"thing":["this","bar",{"name":"alice"}]} # Out: {"all_names":["alice"]} ``` ```bloblang root.text_objects = this.json_path("$.body[?(@.type=='text')]") # In: {"body":[{"type":"image","id":"foo"},{"type":"text","id":"bar"}]} # Out: {"text_objects":[{"id":"bar","type":"text"}]} ``` ### [](#json_schema)json\_schema Checks a [JSON schema](https://json-schema.org/) against a value and returns the value if it matches or throws and error if it does not. #### [](#parameters-62)Parameters | Name | Type | Description | | --- | --- | --- | | schema | string | The schema to check values against. | #### [](#examples-79)Examples ```bloblang root = this.json_schema("""{ "type":"object", "properties":{ "foo":{ "type":"string" } } }""") # In: {"foo":"bar"} # Out: {"foo":"bar"} # In: {"foo":5} # Out: Error("failed assignment (line 1): field `this`: foo invalid type. expected: string, given: integer") ``` In order to load a schema from a file use the `file` function: ```bloblang root = this.json_schema(file(env("BENTHOS_TEST_BLOBLANG_SCHEMA_FILE"))) ``` ### [](#key_values)key\_values Converts an object into an array of key-value pair objects. Each element has a 'key' field and a 'value' field. Order is not guaranteed unless sorted. #### [](#examples-80)Examples ```bloblang root.foo_key_values = this.foo.key_values().sort_by(pair -> pair.key) # In: {"foo":{"bar":1,"baz":2}} # Out: {"foo_key_values":[{"key":"bar","value":1},{"key":"baz","value":2}]} ``` Filter object entries by value: ```bloblang root.large_items = this.items.key_values().filter(pair -> pair.value > 15).map_each(pair -> pair.key) # In: {"items":{"a":5,"b":15,"c":20,"d":3}} # Out: {"large_items":["c"]} ``` ### [](#keys)keys Extracts all keys from an object and returns them as a sorted array. #### [](#examples-81)Examples ```bloblang root.foo_keys = this.foo.keys() # In: {"foo":{"bar":1,"baz":2}} # Out: {"foo_keys":["bar","baz"]} ``` Check if specific keys exist: ```bloblang root.has_id = this.data.keys().contains("id") # In: {"data":{"id":123,"name":"test"}} # Out: {"has_id":true} ``` ### [](#length)length Returns the length of an array, object, or string. #### [](#examples-82)Examples ```bloblang root.foo_len = this.foo.length() # In: {"foo":"hello world"} # Out: {"foo_len":11} ``` ```bloblang root.foo_len = this.foo.length() # In: {"foo":["first","second"]} # Out: {"foo_len":2} # In: {"foo":{"first":"bar","second":"baz"}} # Out: {"foo_len":2} ``` ### [](#map_each)map\_each Applies a mapping to each element of an array or object. #### [](#parameters-63)Parameters | Name | Type | Description | | --- | --- | --- | | query | query expression | A query that will be used to map each element. | #### [](#examples-83)Examples ##### [](#on-arrays-2)On arrays Transforms each array element using a query. Return deleted() to remove an element, or the new value to replace it: ```bloblang root.new_nums = this.nums.map_each(num -> if num < 10 { deleted() } else { num - 10 }) # In: {"nums":[3,11,4,17]} # Out: {"new_nums":[1,7]} ``` ##### [](#on-objects-3)On objects Transforms each object value using a query. The query receives an object with 'key' and 'value' fields for each entry: ```bloblang root.new_dict = this.dict.map_each(item -> item.value.uppercase()) # In: {"dict":{"foo":"hello","bar":"world"}} # Out: {"new_dict":{"bar":"WORLD","foo":"HELLO"}} ``` ### [](#map_each_key)map\_each\_key Transforms object keys using a mapping query. #### [](#parameters-64)Parameters | Name | Type | Description | | --- | --- | --- | | query | query expression | A query that will be used to map each key. | #### [](#examples-84)Examples ```bloblang root.new_dict = this.dict.map_each_key(key -> key.uppercase()) # In: {"dict":{"keya":"hello","keyb":"world"}} # Out: {"new_dict":{"KEYA":"hello","KEYB":"world"}} ``` Conditionally transform keys: ```bloblang root = this.map_each_key(key -> if key.contains("kafka") { "_" + key }) # In: {"amqp_key":"foo","kafka_key":"bar","kafka_topic":"baz"} # Out: {"_kafka_key":"bar","_kafka_topic":"baz","amqp_key":"foo"} ``` ### [](#merge)merge Combines two objects or arrays. When merging objects, conflicting keys create arrays containing both values. Arrays are concatenated. For key override behavior instead, use the assign method. #### [](#parameters-65)Parameters | Name | Type | Description | | --- | --- | --- | | with | unknown | A value to merge the target value with. | #### [](#examples-85)Examples ```bloblang root = this.foo.merge(this.bar) # In: {"foo":{"first_name":"fooer","likes":"bars"},"bar":{"second_name":"barer","likes":"foos"}} # Out: {"first_name":"fooer","likes":["bars","foos"],"second_name":"barer"} ``` Merge arrays: ```bloblang root.combined = this.list1.merge(this.list2) # In: {"list1":["a","b"],"list2":["c","d"]} # Out: {"combined":["a","b","c","d"]} ``` ### [](#patch)patch Applies a changelog (created by the diff method) to the current value, transforming it according to the specified operations. This enables you to synchronize data, replay changes, or implement event sourcing patterns by applying recorded changes to reconstruct state. #### [](#parameters-66)Parameters | Name | Type | Description | | --- | --- | --- | | changelog | unknown | The changelog array to apply. Should be in the format returned by the diff method, containing Type, Path, From, and To fields for each change. | #### [](#examples-86)Examples Apply recorded changes to update an object: ```bloblang root.updated = this.current.patch(this.changelog) # In: {"current":{"name":"Alice","age":30},"changelog":[{"Type":"update","Path":["age"],"From":30,"To":31},{"Type":"create","Path":["city"],"From":null,"To":"NYC"}]} # Out: {"updated":{"age":31,"city":"NYC","name":"Alice"}} ``` Restore previous state by applying inverse changes: ```bloblang root.restored = this.modified.patch(this.reverse_changelog) # In: {"modified":{"timeout":60},"reverse_changelog":[{"Type":"create","Path":["debug"],"From":null,"To":true},{"Type":"update","Path":["timeout"],"From":60,"To":30}]} # Out: {"restored":{"debug":true,"timeout":30}} ``` ### [](#slice)slice Extracts a portion of an array or string. #### [](#parameters-67)Parameters | Name | Type | Description | | --- | --- | --- | | low | integer | The low bound, which is the first element of the selection, or if negative selects from the end. | | high (optional) | integer | An optional high bound. | #### [](#examples-87)Examples ```bloblang root.beginning = this.value.slice(0, 2) root.end = this.value.slice(4) # In: {"value":"foo bar"} # Out: {"beginning":"fo","end":"bar"} ``` A negative low index can be used, indicating an offset from the end of the sequence. If the low index is greater than the length of the sequence then an empty result is returned: ```bloblang root.last_chunk = this.value.slice(-4) root.the_rest = this.value.slice(0, -4) # In: {"value":"foo bar"} # Out: {"last_chunk":" bar","the_rest":"foo"} ``` ```bloblang root.beginning = this.value.slice(0, 2) root.end = this.value.slice(4) # In: {"value":["foo","bar","baz","buz","bev"]} # Out: {"beginning":["foo","bar"],"end":["bev"]} ``` A negative low index can be used, indicating an offset from the end of the sequence. If the low index is greater than the length of the sequence then an empty result is returned: ```bloblang root.last_chunk = this.value.slice(-2) root.the_rest = this.value.slice(0, -2) # In: {"value":["foo","bar","baz","buz","bev"]} # Out: {"last_chunk":["buz","bev"],"the_rest":["foo","bar","baz"]} ``` ### [](#sort)sort Sorts array elements in ascending order. #### [](#parameters-68)Parameters | Name | Type | Description | | --- | --- | --- | | compare (optional) | query expression | An optional query that should explicitly compare elements left and right and provide a boolean result. | #### [](#examples-88)Examples ```bloblang root.sorted = this.foo.sort() # In: {"foo":["bbb","ccc","aaa"]} # Out: {"sorted":["aaa","bbb","ccc"]} ``` Custom comparison for complex objects - return true if left < right: ```bloblang root.sorted = this.foo.sort(item -> item.left.v < item.right.v) # In: {"foo":[{"id":"foo","v":"bbb"},{"id":"bar","v":"ccc"},{"id":"baz","v":"aaa"}]} # Out: {"sorted":[{"id":"baz","v":"aaa"},{"id":"foo","v":"bbb"},{"id":"bar","v":"ccc"}]} ``` ### [](#sort_by)sort\_by Sorts array elements by a specified field or expression. #### [](#parameters-69)Parameters | Name | Type | Description | | --- | --- | --- | | query | query expression | A query to apply to each element that yields a value used for sorting. | #### [](#examples-89)Examples ```bloblang root.sorted = this.foo.sort_by(ele -> ele.id) # In: {"foo":[{"id":"bbb","message":"bar"},{"id":"aaa","message":"foo"},{"id":"ccc","message":"baz"}]} # Out: {"sorted":[{"id":"aaa","message":"foo"},{"id":"bbb","message":"bar"},{"id":"ccc","message":"baz"}]} ``` Sort by numeric field: ```bloblang root.sorted = this.items.sort_by(item -> item.priority) # In: {"items":[{"name":"low","priority":3},{"name":"high","priority":1},{"name":"med","priority":2}]} # Out: {"sorted":[{"name":"high","priority":1},{"name":"med","priority":2},{"name":"low","priority":3}]} ``` ### [](#squash)squash Squashes an array of objects into a single object, where key collisions result in the values being merged (following similar rules as the `.merge()` method). #### [](#examples-90)Examples ```bloblang root.locations = this.locations.map_each(loc -> {loc.state: [loc.name]}).squash() # In: {"locations":[{"name":"Seattle","state":"WA"},{"name":"New York","state":"NY"},{"name":"Bellevue","state":"WA"},{"name":"Olympia","state":"WA"}]} # Out: {"locations":{"NY":["New York"],"WA":["Seattle","Bellevue","Olympia"]}} ``` ### [](#sum)sum Returns the sum of numeric values in an array. #### [](#examples-91)Examples ```bloblang root.sum = this.foo.sum() # In: {"foo":[3,8,4]} # Out: {"sum":15} ``` Works with decimals: ```bloblang root.total = this.prices.sum() # In: {"prices":[10.5,20.25,5.00]} # Out: {"total":35.75} ``` ### [](#unique)unique Returns an array with duplicate elements removed. #### [](#parameters-70)Parameters | Name | Type | Description | | --- | --- | --- | | emit (optional) | query expression | An optional query that can be used in order to yield a value for each element to determine uniqueness. | #### [](#examples-92)Examples ```bloblang root.uniques = this.foo.unique() # In: {"foo":["a","b","a","c"]} # Out: {"uniques":["a","b","c"]} ``` Use a query to determine uniqueness by a field: ```bloblang root.unique_users = this.users.unique(u -> u.id) # In: {"users":[{"id":1,"name":"Alice"},{"id":2,"name":"Bob"},{"id":1,"name":"Alice Duplicate"}]} # Out: {"unique_users":[{"id":1,"name":"Alice"},{"id":2,"name":"Bob"}]} ``` ### [](#values)values Returns an array of all values from an object. #### [](#examples-93)Examples ```bloblang root.foo_vals = this.foo.values().sort() # In: {"foo":{"bar":1,"baz":2}} # Out: {"foo_vals":[1,2]} ``` Find max value in object: ```bloblang root.max = this.scores.values().sort().index(-1) # In: {"scores":{"player1":85,"player2":92,"player3":78}} # Out: {"max":92} ``` ### [](#with)with Returns an object where all but one or more [field path](../../../configuration/field_paths/) arguments are removed. Each path specifies a specific field to be retained from the input object, allowing for nested fields. If a key within a nested path does not exist then it is ignored. #### [](#examples-94)Examples ```bloblang root = this.with("inner.a","inner.c","d") # In: {"inner":{"a":"first","b":"second","c":"third"},"d":"fourth","e":"fifth"} # Out: {"d":"fourth","inner":{"a":"first","c":"third"}} ``` ### [](#without)without Returns an object with specified keys removed. #### [](#examples-95)Examples ```bloblang root = this.without("inner.a","inner.c","d") # In: {"inner":{"a":"first","b":"second","c":"third"},"d":"fourth","e":"fifth"} # Out: {"e":"fifth","inner":{"b":"second"}} ``` Remove sensitive fields: ```bloblang root = this.without("password","ssn","creditCard") # In: {"username":"alice","password":"secret","email":"alice@example.com","ssn":"123-45-6789"} # Out: {"email":"alice@example.com","username":"alice"} ``` ### [](#zip)zip Zip an array value with one or more argument arrays. Each array must match in length. #### [](#examples-96)Examples ```bloblang root.foo = this.foo.zip(this.bar, this.baz) # In: {"foo":["a","b","c"],"bar":[1,2,3],"baz":[4,5,6]} # Out: {"foo":[["a",1,4],["b",2,5],["c",3,6]]} ``` ## [](#parsing)Parsing ### [](#bloblang)bloblang Executes an argument Bloblang mapping on the target. This method can be used in order to execute dynamic mappings. Imports and functions that interact with the environment, such as `file` and `env`, or that access message information directly, such as `content` or `json`, are not enabled for dynamic Bloblang mappings. #### [](#parameters-71)Parameters | Name | Type | Description | | --- | --- | --- | | mapping | string | The mapping to execute. | #### [](#examples-97)Examples ```bloblang root.body = this.body.bloblang(this.mapping) # In: {"body":{"foo":"hello world"},"mapping":"root.foo = this.foo.uppercase()"} # Out: {"body":{"foo":"HELLO WORLD"}} # In: {"body":{"foo":"hello world 2"},"mapping":"root.foo = this.foo.capitalize()"} # Out: {"body":{"foo":"Hello World 2"}} ``` ### [](#format_json)format\_json Formats a value as a JSON string. #### [](#parameters-72)Parameters | Name | Type | Description | | --- | --- | --- | | indent | string | Indentation string. Each element in a JSON object or array will begin on a new, indented line followed by one or more copies of indent according to the indentation nesting. | | no_indent | bool | Disable indentation. | | escape_html | bool | Escape problematic HTML characters. | #### [](#examples-98)Examples ```bloblang root = this.doc.format_json() # In: {"doc":{"foo":"bar"}} # Out: { "foo": "bar" } ``` Pass a string to the `indent` parameter in order to customise the indentation: ```bloblang root = this.format_json(" ") # In: {"doc":{"foo":"bar"}} # Out: { "doc": { "foo": "bar" } } ``` Use the `.string()` method in order to coerce the result into a string: ```bloblang root.doc = this.doc.format_json().string() # In: {"doc":{"foo":"bar"}} # Out: {"doc":"{\n \"foo\": \"bar\"\n}"} ``` Set the `no_indent` parameter to true to disable indentation. The result is equivalent to calling `bytes()`: ```bloblang root = this.doc.format_json(no_indent: true) # In: {"doc":{"foo":"bar"}} # Out: {"foo":"bar"} ``` Escapes problematic HTML characters: ```bloblang root = this.doc.format_json() # In: {"doc":{"email":"foo&bar@benthos.dev","name":"foo>bar"}} # Out: { "email": "foo\u0026bar@benthos.dev", "name": "foo\u003ebar" } ``` Set the `escape_html` parameter to false to disable escaping of problematic HTML characters: ```bloblang root = this.doc.format_json(escape_html: false) # In: {"doc":{"email":"foo&bar@benthos.dev","name":"foo>bar"}} # Out: { "email": "foo&bar@benthos.dev", "name": "foo>bar" } ``` ### [](#format_msgpack)format\_msgpack Serializes structured data into MessagePack binary format. MessagePack is a compact binary serialization that is faster and more space-efficient than JSON, making it ideal for network transmission and storage of structured data. Returns a byte array that can be further encoded as needed. #### [](#examples-99)Examples Serialize object to MessagePack and encode as hex for transmission: ```bloblang root = this.format_msgpack().encode("hex") # In: {"foo":"bar"} # Out: 81a3666f6fa3626172 ``` Serialize data to MessagePack and base64 encode for embedding in JSON: ```bloblang root.msgpack_payload = this.data.format_msgpack().encode("base64") # In: {"data":{"foo":"bar"}} # Out: {"msgpack_payload":"gaNmb2+jYmFy"} ``` ### [](#format_xml)format\_xml Serializes an object into an XML document. Converts structured data to XML format with support for attributes (prefixed with hyphen), custom indentation, and configurable root element. Returns XML as a byte array. #### [](#parameters-73)Parameters | Name | Type | Description | | --- | --- | --- | | indent | string | String to use for each level of indentation (default is 4 spaces). Each nested XML element will be indented by this string. | | no_indent | bool | Disable indentation and newlines to produce compact XML on a single line. | | root_tag (optional) | string | Custom name for the root XML element. By default, the root element name is derived from the first key in the object. | #### [](#examples-100)Examples Serialize object to pretty-printed XML with default indentation: ```bloblang root = this.format_xml() # In: {"foo":{"bar":{"baz":"foo bar baz"}}} # Out: foo bar baz ``` Create compact XML without indentation for smaller message size: ```bloblang root = this.format_xml(no_indent: true) # In: {"foo":{"bar":{"baz":"foo bar baz"}}} # Out: foo bar baz ``` ### [](#format_yaml)format\_yaml Formats a value as a YAML string. #### [](#examples-101)Examples ```bloblang root = this.doc.format_yaml() # In: {"doc":{"foo":"bar"}} # Out: foo: bar ``` Use the `.string()` method in order to coerce the result into a string: ```bloblang root.doc = this.doc.format_yaml().string() # In: {"doc":{"foo":"bar"}} # Out: {"doc":"foo: bar\n"} ``` ### [](#infer_schema)infer\_schema Attempt to infer the schema of a given value. The resulting schema can then be used as an input to schema conversion and enforcement methods. ### [](#parse_csv)parse\_csv Parses CSV data into an array. #### [](#parameters-74)Parameters | Name | Type | Description | | --- | --- | --- | | parse_header_row | bool | Whether to reference the first row as a header row. If set to true the output structure for messages will be an object where field keys are determined by the header row. Otherwise, the output will be an array of row arrays. | | delimiter | string | The delimiter to use for splitting values in each record. It must be a single character. | | lazy_quotes | bool | If set to true, a quote may appear in an unquoted field and a non-doubled quote may appear in a quoted field. | #### [](#examples-102)Examples Parses CSV data with a header row: ```bloblang root.orders = this.orders.parse_csv() # In: {"orders":"foo,bar\nfoo 1,bar 1\nfoo 2,bar 2"} # Out: {"orders":[{"bar":"bar 1","foo":"foo 1"},{"bar":"bar 2","foo":"foo 2"}]} ``` Parses CSV data without a header row: ```bloblang root.orders = this.orders.parse_csv(false) # In: {"orders":"foo 1,bar 1\nfoo 2,bar 2"} # Out: {"orders":[["foo 1","bar 1"],["foo 2","bar 2"]]} ``` Parses CSV data delimited by dots: ```bloblang root.orders = this.orders.parse_csv(delimiter:".") # In: {"orders":"foo.bar\nfoo 1.bar 1\nfoo 2.bar 2"} # Out: {"orders":[{"bar":"bar 1","foo":"foo 1"},{"bar":"bar 2","foo":"foo 2"}]} ``` Parses CSV data containing a quote in an unquoted field: ```bloblang root.orders = this.orders.parse_csv(lazy_quotes:true) # In: {"orders":"foo,bar\nfoo 1,bar 1\nfoo\" \"2,bar\" \"2"} # Out: {"orders":[{"bar":"bar 1","foo":"foo 1"},{"bar":"bar\" \"2","foo":"foo\" \"2"}]} ``` ### [](#parse_form_url_encoded)parse\_form\_url\_encoded Attempts to parse a url-encoded query string (from an x-www-form-urlencoded request body) and returns a structured result. #### [](#examples-103)Examples ```bloblang root.values = this.body.parse_form_url_encoded() # In: {"body":"noise=meow&animal=cat&fur=orange&fur=fluffy"} # Out: {"values":{"animal":"cat","fur":["orange","fluffy"],"noise":"meow"}} ``` ### [](#parse_json)parse\_json Parses a JSON string into a structured value. #### [](#parameters-75)Parameters | Name | Type | Description | | --- | --- | --- | | use_number (optional) | bool | An optional flag that when set makes parsing numbers as json.Number instead of the default float64. | #### [](#examples-104)Examples ```bloblang root.doc = this.doc.parse_json() # In: {"doc":"{\"foo\":\"bar\"}"} # Out: {"doc":{"foo":"bar"}} ``` ```bloblang root.doc = this.doc.parse_json(use_number: true) # In: {"doc":"{\"foo\":\"11380878173205700000000000000000000000000000000\"}"} # Out: {"doc":{"foo":"11380878173205700000000000000000000000000000000"}} ``` ### [](#parse_logfmt)parse\_logfmt Parses logfmt formatted data into an object. #### [](#examples-105)Examples ```bloblang root = this.msg.parse_logfmt() # In: {"msg":"level=info msg=\"hello world\" dur=1.5s"} # Out: {"dur":"1.5s","level":"info","msg":"hello world"} ``` ### [](#parse_msgpack)parse\_msgpack Parses MessagePack binary data into a structured object. MessagePack is an efficient binary serialization format that is more compact than JSON while maintaining similar data structures. Commonly used for high-performance APIs and data interchange between microservices. #### [](#examples-106)Examples Parse MessagePack data from hex-encoded content: ```bloblang root = content().decode("hex").parse_msgpack() # In: 81a3666f6fa3626172 # Out: {"foo":"bar"} ``` Parse MessagePack from base64-encoded field: ```bloblang root.decoded = this.msgpack_data.decode("base64").parse_msgpack() # In: {"msgpack_data":"gaNmb2+jYmFy"} # Out: {"decoded":{"foo":"bar"}} ``` ### [](#parse_parquet)parse\_parquet Parses Apache Parquet binary data into an array of objects. Parquet is a columnar storage format optimized for analytics, commonly used with big data systems like Apache Spark, Hive, and cloud data warehouses. Each row in the Parquet file becomes an object in the output array. #### [](#parameters-76)Parameters | Name | Type | Description | | --- | --- | --- | | byte_array_as_string | bool | Deprecated: This parameter is no longer used. | #### [](#examples-107)Examples Parse Parquet file data into structured objects: ```bloblang root.records = content().parse_parquet() ``` Process Parquet data from a field and extract specific columns: ```bloblang root.users = this.parquet_data.parse_parquet().map_each(row -> {"name": row.name, "email": row.email}) ``` ### [](#parse_url)parse\_url Attempts to parse a URL from a string value, returning a structured result that describes the various facets of the URL. The fields returned within the structured result roughly follow [https://pkg.go.dev/net/url#URL](https://pkg.go.dev/net/url#URL), and may be expanded in future in order to present more information. #### [](#examples-108)Examples ```bloblang root.foo_url = this.foo_url.parse_url() # In: {"foo_url":"https://docs.redpanda.com/redpanda-connect/guides/bloblang/about/"} # Out: {"foo_url":{"fragment":"","host":"docs.redpanda.com","opaque":"","path":"/redpanda-connect/guides/bloblang/about/","raw_fragment":"","raw_path":"","raw_query":"","scheme":"https"}} ``` ```bloblang root.username = this.url.parse_url().user.name | "unknown" # In: {"url":"amqp://foo:bar@127.0.0.1:5672/"} # Out: {"username":"foo"} # In: {"url":"redis://localhost:6379"} # Out: {"username":"unknown"} ``` ### [](#parse_xml)parse\_xml Parses an XML document into a structured object. Converts XML elements to JSON-like objects following these rules: - Element attributes are prefixed with a hyphen (e.g., `-id` for an `id` attribute) - Elements with both attributes and text content store the text in a `#text` field - Repeated elements become arrays - XML comments, directives, and processing instructions are ignored - Optionally cast numeric and boolean strings to their proper types. #### [](#parameters-77)Parameters | Name | Type | Description | | --- | --- | --- | | cast (optional) | bool | Whether to automatically cast numeric and boolean string values to their proper types. When false, all values remain as strings. | #### [](#examples-109)Examples Parse XML document into object structure: ```bloblang root.doc = this.doc.parse_xml() # In: {"doc":"This is a titleThis is some content"} # Out: {"doc":{"root":{"content":"This is some content","title":"This is a title"}}} ``` Parse XML with type casting enabled to convert strings to numbers and booleans: ```bloblang root.doc = this.doc.parse_xml(cast: true) # In: {"doc":"This is a title123True"} # Out: {"doc":{"root":{"bool":true,"number":{"#text":123,"-id":99},"title":"This is a title"}}} ``` ### [](#parse_yaml)parse\_yaml Parses a YAML string into a structured value. #### [](#examples-110)Examples ```bloblang root.doc = this.doc.parse_yaml() # In: {"doc":"foo: bar"} # Out: {"doc":{"foo":"bar"}} ``` ## [](#regular-expressions)Regular expressions ### [](#re_find_all)re\_find\_all Finds all matches of a regular expression in a string. #### [](#parameters-78)Parameters | Name | Type | Description | | --- | --- | --- | | pattern | string | The pattern to match against. | #### [](#examples-111)Examples ```bloblang root.matches = this.value.re_find_all("a.") # In: {"value":"paranormal"} # Out: {"matches":["ar","an","al"]} ``` ```bloblang root.numbers = this.text.re_find_all("[0-9]+") # In: {"text":"I have 2 apples and 15 oranges"} # Out: {"numbers":["2","15"]} ``` ### [](#re_find_all_object)re\_find\_all\_object Finds all regex matches as objects with named groups. #### [](#parameters-79)Parameters | Name | Type | Description | | --- | --- | --- | | pattern | string | The pattern to match against. | #### [](#examples-112)Examples ```bloblang root.matches = this.value.re_find_all_object("a(?Px*)b") # In: {"value":"-axxb-ab-"} # Out: {"matches":[{"0":"axxb","foo":"xx"},{"0":"ab","foo":""}]} ``` ```bloblang root.matches = this.value.re_find_all_object("(?m)(?P\\w+):\\s+(?P\\w+)$") # In: {"value":"option1: value1\noption2: value2\noption3: value3"} # Out: {"matches":[{"0":"option1: value1","key":"option1","value":"value1"},{"0":"option2: value2","key":"option2","value":"value2"},{"0":"option3: value3","key":"option3","value":"value3"}]} ``` ### [](#re_find_all_submatch)re\_find\_all\_submatch Finds all regex matches with capture groups. #### [](#parameters-80)Parameters | Name | Type | Description | | --- | --- | --- | | pattern | string | The pattern to match against. | #### [](#examples-113)Examples ```bloblang root.matches = this.value.re_find_all_submatch("a(x*)b") # In: {"value":"-axxb-ab-"} # Out: {"matches":[["axxb","xx"],["ab",""]]} ``` ```bloblang root.emails = this.text.re_find_all_submatch("(\\w+)@(\\w+\\.\\w+)") # In: {"text":"Contact: alice@example.com or bob@test.org"} # Out: {"emails":[["alice@example.com","alice","example.com"],["bob@test.org","bob","test.org"]]} ``` ### [](#re_find_object)re\_find\_object Finds the first regex match as an object with named groups. #### [](#parameters-81)Parameters | Name | Type | Description | | --- | --- | --- | | pattern | string | The pattern to match against. | #### [](#examples-114)Examples ```bloblang root.matches = this.value.re_find_object("a(?Px*)b") # In: {"value":"-axxb-ab-"} # Out: {"matches":{"0":"axxb","foo":"xx"}} ``` ```bloblang root.matches = this.value.re_find_object("(?P\\w+):\\s+(?P\\w+)") # In: {"value":"option1: value1"} # Out: {"matches":{"0":"option1: value1","key":"option1","value":"value1"}} ``` ### [](#re_match)re\_match Tests if a string matches a regular expression. #### [](#parameters-82)Parameters | Name | Type | Description | | --- | --- | --- | | pattern | string | The pattern to match against. | #### [](#examples-115)Examples ```bloblang root.matches = this.value.re_match("[0-9]") # In: {"value":"there are 10 puppies"} # Out: {"matches":true} # In: {"value":"there are ten puppies"} # Out: {"matches":false} ``` ### [](#re_replace)re\_replace Replaces all regex matches with a replacement string that can reference capture groups using `$1`, `$2`, etc. Use for pattern-based transformations or data reformatting. #### [](#parameters-83)Parameters | Name | Type | Description | | --- | --- | --- | | pattern | string | The pattern to match against. | | value | string | The value to replace with. | ### [](#re_replace_all)re\_replace\_all Replaces all regex matches with a replacement string. #### [](#parameters-84)Parameters | Name | Type | Description | | --- | --- | --- | | pattern | string | The pattern to match against. | | value | string | The value to replace with. | #### [](#examples-116)Examples ```bloblang root.new_value = this.value.re_replace_all("ADD ([0-9]+)","+($1)") # In: {"value":"foo ADD 70"} # Out: {"new_value":"foo +(70)"} ``` ```bloblang root.masked = this.email.re_replace_all("(\\w{2})\\w+@", "$1***@") # In: {"email":"alice@example.com"} # Out: {"masked":"al***@example.com"} ``` ## [](#sql)SQL ### [](#vector)vector Converts an array of numbers into a vector type suitable for insertion into SQL databases with vector/embedding support. This is commonly used with PostgreSQL’s pgvector extension for storing and querying machine learning embeddings, enabling similarity search and vector operations in your database. #### [](#examples-117)Examples Convert embeddings array to vector for pgvector storage: ```bloblang root.embedding = this.embeddings.vector() root.text = this.text ``` Process ML model output into database-ready vector format: ```bloblang root.doc_id = this.id root.vector_embedding = this.model_output.map_each(num -> num.number()).vector() ``` ## [](#string-manipulation)String manipulation ### [](#capitalize)capitalize Converts a string to title case with Unicode letter mapping. #### [](#examples-118)Examples ```bloblang root.title = this.title.capitalize() # In: {"title":"the foo bar"} # Out: {"title":"The Foo Bar"} ``` ```bloblang root.name = this.name.capitalize() # In: {"name":"alice smith"} # Out: {"name":"Alice Smith"} ``` ### [](#compare_argon2)compare\_argon2 Checks whether a string matches a hashed secret using Argon2. #### [](#parameters-85)Parameters | Name | Type | Description | | --- | --- | --- | | hashed_secret | string | The hashed secret to compare with the input. This must be a fully-qualified string which encodes the Argon2 options used to generate the hash. | #### [](#examples-119)Examples ```bloblang root.match = this.secret.compare_argon2("$argon2id$v=19$m=4096,t=3,p=1$c2FsdHktbWNzYWx0ZmFjZQ$RMUMwgtS32/mbszd+ke4o4Ej1jFpYiUqY6MHWa69X7Y") # In: {"secret":"there-are-many-blobs-in-the-sea"} # Out: {"match":true} ``` ```bloblang root.match = this.secret.compare_argon2("$argon2id$v=19$m=4096,t=3,p=1$c2FsdHktbWNzYWx0ZmFjZQ$RMUMwgtS32/mbszd+ke4o4Ej1jFpYiUqY6MHWa69X7Y") # In: {"secret":"will-i-ever-find-love"} # Out: {"match":false} ``` ### [](#compare_bcrypt)compare\_bcrypt Checks whether a string matches a hashed secret using bcrypt. #### [](#parameters-86)Parameters | Name | Type | Description | | --- | --- | --- | | hashed_secret | string | The hashed secret value to compare with the input. | #### [](#examples-120)Examples ```bloblang root.match = this.secret.compare_bcrypt("$2y$10$Dtnt5NNzVtMCOZONT705tOcS8It6krJX8bEjnDJnwxiFKsz1C.3Ay") # In: {"secret":"there-are-many-blobs-in-the-sea"} # Out: {"match":true} ``` ```bloblang root.match = this.secret.compare_bcrypt("$2y$10$Dtnt5NNzVtMCOZONT705tOcS8It6krJX8bEjnDJnwxiFKsz1C.3Ay") # In: {"secret":"will-i-ever-find-love"} # Out: {"match":false} ``` ### [](#contains-2)contains Tests if an array or object contains a value. #### [](#parameters-87)Parameters | Name | Type | Description | | --- | --- | --- | | value | unknown | A value to test against elements of the target. | #### [](#examples-121)Examples ```bloblang root.has_foo = this.thing.contains("foo") # In: {"thing":["this","foo","that"]} # Out: {"has_foo":true} # In: {"thing":["this","bar","that"]} # Out: {"has_foo":false} ``` ```bloblang root.has_bar = this.thing.contains(20) # In: {"thing":[10.3,20.0,"huh",3]} # Out: {"has_bar":true} # In: {"thing":[2,3,40,67]} # Out: {"has_bar":false} ``` ```bloblang root.has_foo = this.thing.contains("foo") # In: {"thing":"this foo that"} # Out: {"has_foo":true} # In: {"thing":"this bar that"} # Out: {"has_foo":false} ``` ### [](#escape_html)escape\_html Escapes HTML special characters. #### [](#examples-122)Examples ```bloblang root.escaped = this.value.escape_html() # In: {"value":"foo & bar"} # Out: {"escaped":"foo & bar"} ``` ```bloblang root.safe_html = this.user_input.escape_html() # In: {"user_input":""} # Out: {"safe_html":"<script>alert('xss')</script>"} ``` ### [](#escape_url_path)escape\_url\_path Escapes a string for use in URL paths. #### [](#examples-123)Examples ```bloblang root.escaped = this.value.escape_url_path() # In: {"value":"foo & bar"} # Out: {"escaped":"foo%20&%20bar"} ``` ```bloblang root.url = "https://example.com/docs/" + this.path.escape_url_path() # In: {"path":"my document.pdf"} # Out: {"url":"https://example.com/docs/my%20document.pdf"} ``` ### [](#escape_url_query)escape\_url\_query Escapes a string for use in URL query parameters. #### [](#examples-124)Examples ```bloblang root.escaped = this.value.escape_url_query() # In: {"value":"foo & bar"} # Out: {"escaped":"foo+%26+bar"} ``` ```bloblang root.url = "https://example.com?search=" + this.query.escape_url_query() # In: {"query":"hello world!"} # Out: {"url":"https://example.com?search=hello+world%21"} ``` ### [](#filepath_join)filepath\_join Joins filepath components into a single path. #### [](#examples-125)Examples ```bloblang root.path = this.path_elements.filepath_join() # In: {"path_elements":["/foo/","bar.txt"]} # Out: {"path":"/foo/bar.txt"} ``` ### [](#filepath_split)filepath\_split Splits a filepath into directory and filename components. #### [](#examples-126)Examples ```bloblang root.path_sep = this.path.filepath_split() # In: {"path":"/foo/bar.txt"} # Out: {"path_sep":["/foo/","bar.txt"]} # In: {"path":"baz.txt"} # Out: {"path_sep":["","baz.txt"]} ``` ### [](#format)format Formats a value using a specified format string. #### [](#examples-127)Examples ```bloblang root.foo = "%s(%v): %v".format(this.name, this.age, this.fingers) # In: {"name":"lance","age":37,"fingers":13} # Out: {"foo":"lance(37): 13"} ``` ```bloblang root.message = "User %s has %v points".format(this.username, this.score) # In: {"username":"alice","score":100} # Out: {"message":"User alice has 100 points"} ``` ### [](#has_prefix)has\_prefix Tests if a string starts with a specified prefix. #### [](#parameters-88)Parameters | Name | Type | Description | | --- | --- | --- | | value | string | The string to test. | #### [](#examples-128)Examples ```bloblang root.t1 = this.v1.has_prefix("foo") root.t2 = this.v2.has_prefix("foo") # In: {"v1":"foobar","v2":"barfoo"} # Out: {"t1":true,"t2":false} ``` ### [](#has_suffix)has\_suffix Tests if a string ends with a specified suffix. #### [](#parameters-89)Parameters | Name | Type | Description | | --- | --- | --- | | value | string | The string to test. | #### [](#examples-129)Examples ```bloblang root.t1 = this.v1.has_suffix("foo") root.t2 = this.v2.has_suffix("foo") # In: {"v1":"foobar","v2":"barfoo"} # Out: {"t1":false,"t2":true} ``` ### [](#index_of)index\_of Returns the index of the first occurrence of a substring. #### [](#parameters-90)Parameters | Name | Type | Description | | --- | --- | --- | | value | string | A string to search for. | #### [](#examples-130)Examples ```bloblang root.index = this.thing.index_of("bar") # In: {"thing":"foobar"} # Out: {"index":3} ``` ```bloblang root.index = content().index_of("meow") # In: the cat meowed, the dog woofed # Out: {"index":8} ``` ### [](#length-2)length Returns the length of an array, object, or string. #### [](#examples-131)Examples ```bloblang root.foo_len = this.foo.length() # In: {"foo":"hello world"} # Out: {"foo_len":11} ``` ```bloblang root.foo_len = this.foo.length() # In: {"foo":["first","second"]} # Out: {"foo_len":2} # In: {"foo":{"first":"bar","second":"baz"}} # Out: {"foo_len":2} ``` ### [](#lowercase)lowercase Converts all letters in a string to lowercase. #### [](#examples-132)Examples ```bloblang root.foo = this.foo.lowercase() # In: {"foo":"HELLO WORLD"} # Out: {"foo":"hello world"} ``` ```bloblang root.email = this.user_email.lowercase() # In: {"user_email":"User@Example.COM"} # Out: {"email":"user@example.com"} ``` ### [](#quote)quote Wraps a string in double quotes and escapes special characters. #### [](#examples-133)Examples ```bloblang root.quoted = this.thing.quote() # In: {"thing":"foo\nbar"} # Out: {"quoted":"\"foo\\nbar\""} ``` ```bloblang root.literal = this.text.quote() # In: {"text":"hello\tworld"} # Out: {"literal":"\"hello\\tworld\""} ``` ### [](#repeat)repeat Creates a string by repeating the input a specified number of times. #### [](#parameters-91)Parameters | Name | Type | Description | | --- | --- | --- | | count | integer | The number of times to repeat the string. | #### [](#examples-134)Examples ```bloblang root.repeated = this.name.repeat(3) root.not_repeated = this.name.repeat(0) # In: {"name":"bob"} # Out: {"not_repeated":"","repeated":"bobbobbob"} ``` ```bloblang root.separator = "-".repeat(10) # In: {} # Out: {"separator":"----------"} ``` ### [](#replace)replace Replaces all occurrences of a substring with another string. Use for text transformation, cleaning data, or normalizing strings. #### [](#parameters-92)Parameters | Name | Type | Description | | --- | --- | --- | | old | string | A string to match against. | | new | string | A string to replace with. | ### [](#replace_all)replace\_all Replaces all occurrences of a substring with another. #### [](#parameters-93)Parameters | Name | Type | Description | | --- | --- | --- | | old | string | A string to match against. | | new | string | A string to replace with. | #### [](#examples-135)Examples ```bloblang root.new_value = this.value.replace_all("foo","dog") # In: {"value":"The foo ate my homework"} # Out: {"new_value":"The dog ate my homework"} ``` ```bloblang root.clean = this.text.replace_all(" ", " ") # In: {"text":"hello world foo"} # Out: {"clean":"hello world foo"} ``` ### [](#replace_all_many)replace\_all\_many Performs multiple find-and-replace operations in sequence. #### [](#parameters-94)Parameters | Name | Type | Description | | --- | --- | --- | | values | array | An array of values, each even value will be replaced with the following odd value. | #### [](#examples-136)Examples ```bloblang root.new_value = this.value.replace_all_many([ "", "<b>", "", "</b>", "", "<i>", "", "</i>", ]) # In: {"value":"Hello World"} # Out: {"new_value":"<i>Hello</i> <b>World</b>"} ``` ### [](#replace_many)replace\_many Performs multiple find-and-replace operations in sequence using an array of `[old, new]` pairs. More efficient than chaining multiple `replace_all` calls. Use for bulk text transformations. #### [](#parameters-95)Parameters | Name | Type | Description | | --- | --- | --- | | values | array | An array of values, each even value will be replaced with the following odd value. | ### [](#reverse)reverse Reverses the order of characters in a string. #### [](#examples-137)Examples ```bloblang root.reversed = this.thing.reverse() # In: {"thing":"backwards"} # Out: {"reversed":"sdrawkcab"} ``` ```bloblang root = content().reverse() # In: {"thing":"backwards"} # Out: }"sdrawkcab":"gniht"{ ``` ### [](#slice-2)slice Extracts a portion of an array or string. #### [](#parameters-96)Parameters | Name | Type | Description | | --- | --- | --- | | low | integer | The low bound, which is the first element of the selection, or if negative selects from the end. | | high (optional) | integer | An optional high bound. | #### [](#examples-138)Examples ```bloblang root.beginning = this.value.slice(0, 2) root.end = this.value.slice(4) # In: {"value":"foo bar"} # Out: {"beginning":"fo","end":"bar"} ``` A negative low index can be used, indicating an offset from the end of the sequence. If the low index is greater than the length of the sequence then an empty result is returned: ```bloblang root.last_chunk = this.value.slice(-4) root.the_rest = this.value.slice(0, -4) # In: {"value":"foo bar"} # Out: {"last_chunk":" bar","the_rest":"foo"} ``` ```bloblang root.beginning = this.value.slice(0, 2) root.end = this.value.slice(4) # In: {"value":["foo","bar","baz","buz","bev"]} # Out: {"beginning":["foo","bar"],"end":["bev"]} ``` A negative low index can be used, indicating an offset from the end of the sequence. If the low index is greater than the length of the sequence then an empty result is returned: ```bloblang root.last_chunk = this.value.slice(-2) root.the_rest = this.value.slice(0, -2) # In: {"value":["foo","bar","baz","buz","bev"]} # Out: {"last_chunk":["buz","bev"],"the_rest":["foo","bar","baz"]} ``` ### [](#slug)slug Converts a string into a URL-friendly slug by replacing spaces with hyphens, removing special characters, and converting to lowercase. Supports multiple languages for proper transliteration of non-ASCII characters. #### [](#parameters-97)Parameters | Name | Type | Description | | --- | --- | --- | | lang (optional) | string | | #### [](#examples-139)Examples Create a URL-friendly slug from a string with special characters: ```bloblang root.slug = this.title.slug() # In: {"title":"Hello World! Welcome to Redpanda Connect"} # Out: {"slug":"hello-world-welcome-to-redpanda-connect"} ``` Create a slug preserving French language rules: ```bloblang root.slug = this.title.slug("fr") # In: {"title":"Café & Restaurant"} # Out: {"slug":"cafe-et-restaurant"} ``` ### [](#split)split Splits a string into an array of substrings. #### [](#parameters-98)Parameters | Name | Type | Description | | --- | --- | --- | | delimiter | string | The delimiter to split with. | | empty_as_null | bool | To treat empty substrings as null values | #### [](#examples-140)Examples ```bloblang root.new_value = this.value.split(",") # In: {"value":"foo,bar,baz"} # Out: {"new_value":["foo","bar","baz"]} ``` ```bloblang root.new_value = this.value.split(",", true) # In: {"value":"foo,,qux"} # Out: {"new_value":["foo",null,"qux"]} ``` ```bloblang root.words = this.sentence.split(" ") # In: {"sentence":"hello world from bloblang"} # Out: {"words":["hello","world","from","bloblang"]} ``` ### [](#strip_html)strip\_html Removes HTML tags from a string, returning only the text content. Useful for extracting plain text from HTML documents, sanitizing user input, or preparing content for text analysis. Optionally preserves specific HTML elements while stripping all others. #### [](#parameters-99)Parameters | Name | Type | Description | | --- | --- | --- | | preserve (optional) | unknown | Optional array of HTML element names to preserve (e.g., ["strong", "em", "a"]). All other HTML tags will be removed. | #### [](#examples-141)Examples Extract plain text from HTML content: ```bloblang root.plain_text = this.html_content.strip_html() # In: {"html_content":"

Welcome to Redpanda Connect!

"} # Out: {"plain_text":"Welcome to Redpanda Connect!"} ``` Preserve specific HTML elements while removing others: ```bloblang root.sanitized = this.html.strip_html(["strong", "em"]) # In: {"html":"

Some bold and italic text with a

"} # Out: {"sanitized":"Some bold and italic text with a "} ``` ### [](#trim)trim Removes leading and trailing characters from a string. #### [](#parameters-100)Parameters | Name | Type | Description | | --- | --- | --- | | cutset (optional) | string | An optional string of characters to trim from the target value. | #### [](#examples-142)Examples ```bloblang root.title = this.title.trim("!?") root.description = this.description.trim() # In: {"description":" something happened and its amazing! ","title":"!!!watch out!?"} # Out: {"description":"something happened and its amazing!","title":"watch out"} ``` ### [](#trim_prefix)trim\_prefix Removes a specified prefix from the beginning of a string. #### [](#parameters-101)Parameters | Name | Type | Description | | --- | --- | --- | | prefix | string | The leading prefix substring to trim from the string. | #### [](#examples-143)Examples ```bloblang root.name = this.name.trim_prefix("foobar_") root.description = this.description.trim_prefix("foobar_") # In: {"description":"unchanged","name":"foobar_blobton"} # Out: {"description":"unchanged","name":"blobton"} ``` ### [](#trim_suffix)trim\_suffix Removes a specified suffix from the end of a string. #### [](#parameters-102)Parameters | Name | Type | Description | | --- | --- | --- | | suffix | string | The trailing suffix substring to trim from the string. | #### [](#examples-144)Examples ```bloblang root.name = this.name.trim_suffix("_foobar") root.description = this.description.trim_suffix("_foobar") # In: {"description":"unchanged","name":"blobton_foobar"} # Out: {"description":"unchanged","name":"blobton"} ``` ### [](#unescape_html)unescape\_html Converts HTML entities back to their original characters. #### [](#examples-145)Examples ```bloblang root.unescaped = this.value.unescape_html() # In: {"value":"foo & bar"} # Out: {"unescaped":"foo & bar"} ``` ```bloblang root.text = this.html.unescape_html() # In: {"html":"<p>Hello & goodbye</p>"} # Out: {"text":"

Hello & goodbye

"} ``` ### [](#unescape_url_path)unescape\_url\_path Unescapes URL path encoding. #### [](#examples-146)Examples ```bloblang root.unescaped = this.value.unescape_url_path() # In: {"value":"foo%20&%20bar"} # Out: {"unescaped":"foo & bar"} ``` ```bloblang root.filename = this.path.unescape_url_path() # In: {"path":"my%20document.pdf"} # Out: {"filename":"my document.pdf"} ``` ### [](#unescape_url_query)unescape\_url\_query Unescapes URL query parameter encoding. #### [](#examples-147)Examples ```bloblang root.unescaped = this.value.unescape_url_query() # In: {"value":"foo+%26+bar"} # Out: {"unescaped":"foo & bar"} ``` ```bloblang root.search = this.param.unescape_url_query() # In: {"param":"hello+world%21"} # Out: {"search":"hello world!"} ``` ### [](#unicode_segments)unicode\_segments Splits text into segments based on Unicode text segmentation rules. Returns an array of strings representing individual graphemes (visual characters), words (including punctuation and whitespace), or sentences. Handles complex Unicode correctly, including emoji with skin tone modifiers and zero-width joiners. #### [](#parameters-103)Parameters | Name | Type | Description | | --- | --- | --- | | segmentation_type | string | Type of segmentation: "grapheme", "word", or "sentence" | #### [](#examples-148)Examples Split text into sentences (preserves trailing spaces): ```bloblang root.sentences = this.text.unicode_segments("sentence") # In: {"text":"Hello world. How are you?"} # Out: {"sentences":["Hello world. ","How are you?"]} ``` Split text into grapheme clusters (handles complex emoji correctly): ```bloblang root.graphemes = this.emoji.unicode_segments("grapheme") # In: {"emoji":"👨‍👩‍👧‍👦❤️"} # Out: {"graphemes":["👨‍👩‍👧‍👦","❤️"]} ``` ### [](#unquote)unquote Removes surrounding quotes and interprets escape sequences. #### [](#examples-149)Examples ```bloblang root.unquoted = this.thing.unquote() # In: {"thing":"\"foo\\nbar\""} # Out: {"unquoted":"foo\nbar"} ``` ```bloblang root.text = this.literal.unquote() # In: {"literal":"\"hello\\tworld\""} # Out: {"text":"hello\tworld"} ``` ### [](#uppercase)uppercase Converts all letters in a string to uppercase. #### [](#examples-150)Examples ```bloblang root.foo = this.foo.uppercase() # In: {"foo":"hello world"} # Out: {"foo":"HELLO WORLD"} ``` ```bloblang root.code = this.product_code.uppercase() # In: {"product_code":"abc-123"} # Out: {"code":"ABC-123"} ``` ## [](#timestamp-manipulation)Timestamp manipulation ### [](#parse_duration)parse\_duration Parses a Go-style duration string into nanoseconds. A duration string is a signed sequence of decimal numbers with unit suffixes like "300ms", "-1.5h", or "2h45m". Valid units: "ns", "us" (or "µs"), "ms", "s", "m", "h". #### [](#examples-151)Examples Parse microseconds to nanoseconds: ```bloblang root.delay_for_ns = this.delay_for.parse_duration() # In: {"delay_for":"50us"} # Out: {"delay_for_ns":50000} ``` Parse hours to seconds: ```bloblang root.delay_for_s = this.delay_for.parse_duration() / 1000000000 # In: {"delay_for":"2h"} # Out: {"delay_for_s":7200} ``` ### [](#parse_duration_iso8601)parse\_duration\_iso8601 Parses an ISO 8601 duration string into nanoseconds. Format: "P\[n\]Y\[n\]M\[n\]DT\[n\]H\[n\]M\[n\]S" or "P\[n\]W". Example: "P3Y6M4DT12H30M5S" means 3 years, 6 months, 4 days, 12 hours, 30 minutes, 5 seconds. Supports fractional seconds with full precision (not just one decimal place). #### [](#examples-152)Examples Parse complex ISO 8601 duration to nanoseconds: ```bloblang root.delay_for_ns = this.delay_for.parse_duration_iso8601() # In: {"delay_for":"P3Y6M4DT12H30M5S"} # Out: {"delay_for_ns":110839937000000000} ``` Parse hours to seconds: ```bloblang root.delay_for_s = this.delay_for.parse_duration_iso8601() / 1000000000 # In: {"delay_for":"PT2H"} # Out: {"delay_for_s":7200} ``` ### [](#ts_add_iso8601)ts\_add\_iso8601 Adds an ISO 8601 duration to a timestamp with calendar-aware precision for years, months, and days. Useful when you need to add durations that account for variable month lengths or leap years. #### [](#parameters-104)Parameters | Name | Type | Description | | --- | --- | --- | | duration | string | Duration in ISO 8601 format (e.g., "P1Y2M3D" for 1 year, 2 months, 3 days) | #### [](#examples-153)Examples Add one year to a timestamp: ```bloblang root.next_year = this.created_at.ts_add_iso8601("P1Y") # In: {"created_at":"2020-08-14T05:54:23Z"} # Out: {"next_year":"2021-08-14T05:54:23Z"} ``` Add a complex duration with multiple units: ```bloblang root.future_date = this.created_at.ts_add_iso8601("P1Y2M3DT4H5M6S") # In: {"created_at":"2020-01-01T00:00:00Z"} # Out: {"future_date":"2021-03-04T04:05:06Z"} ``` ### [](#ts_format)ts\_format Formats a timestamp as a string using Go’s reference time format. Defaults to RFC 3339 if no format specified. The format uses "Mon Jan 2 15:04:05 -0700 MST 2006" as a reference. Accepts unix timestamps (with decimal precision) or RFC 3339 strings. Use ts\_strftime for strftime-style formats. #### [](#parameters-105)Parameters | Name | Type | Description | | --- | --- | --- | | format | string | The output format using Go’s reference time. | | tz (optional) | string | Optional timezone (e.g., 'UTC', 'America/New_York'). Defaults to input timezone or local time for unix timestamps. | #### [](#examples-154)Examples Format timestamp with custom format: ```bloblang root.something_at = this.created_at.ts_format("2006-Jan-02 15:04:05") # In: {"created_at":"2020-08-14T11:50:26.371Z"} # Out: {"something_at":"2020-Aug-14 11:50:26"} ``` Format unix timestamp with timezone specification: ```bloblang root.something_at = this.created_at.ts_format(format: "2006-Jan-02 15:04:05", tz: "UTC") # In: {"created_at":1597405526} # Out: {"something_at":"2020-Aug-14 11:45:26"} ``` ### [](#ts_parse)ts\_parse Parses a timestamp string using Go’s reference time format and outputs a timestamp object. The format uses "Mon Jan 2 15:04:05 -0700 MST 2006" as a reference - show how this reference time would appear in your format. Use ts\_strptime for strftime-style formats instead. #### [](#parameters-106)Parameters | Name | Type | Description | | --- | --- | --- | | format | string | The format of the input string using Go’s reference time. | #### [](#examples-155)Examples Parse a date with abbreviated month name: ```bloblang root.doc.timestamp = this.doc.timestamp.ts_parse("2006-Jan-02") # In: {"doc":{"timestamp":"2020-Aug-14"}} # Out: {"doc":{"timestamp":"2020-08-14T00:00:00Z"}} ``` Parse a custom datetime format: ```bloblang root.parsed = this.timestamp.ts_parse("Jan 2, 2006 at 3:04pm (MST)") # In: {"timestamp":"Aug 14, 2020 at 5:54am (UTC)"} # Out: {"parsed":"2020-08-14T05:54:00Z"} ``` ### [](#ts_round)ts\_round Rounds a timestamp to the nearest multiple of the specified duration. Halfway values round up. Accepts unix timestamps (seconds with optional decimal precision) or RFC 3339 formatted strings. #### [](#parameters-107)Parameters | Name | Type | Description | | --- | --- | --- | | duration | integer | A duration measured in nanoseconds to round by. | #### [](#examples-156)Examples Round timestamp to the nearest hour: ```bloblang root.created_at_hour = this.created_at.ts_round("1h".parse_duration()) # In: {"created_at":"2020-08-14T05:54:23Z"} # Out: {"created_at_hour":"2020-08-14T06:00:00Z"} ``` Round timestamp to the nearest minute: ```bloblang root.created_at_minute = this.created_at.ts_round("1m".parse_duration()) # In: {"created_at":"2020-08-14T05:54:23Z"} # Out: {"created_at_minute":"2020-08-14T05:54:00Z"} ``` ### [](#ts_strftime)ts\_strftime Formats a timestamp as a string using strptime format specifiers (like %Y, %m, %d). Accepts unix timestamps (with decimal precision) or RFC 3339 strings. Supports %f for microseconds. Use ts\_format for Go-style reference time formats. #### [](#parameters-108)Parameters | Name | Type | Description | | --- | --- | --- | | format | string | The output format using strptime specifiers. | | tz (optional) | string | Optional timezone. Defaults to input timezone or local time for unix timestamps. | #### [](#examples-157)Examples Format timestamp with strftime specifiers: ```bloblang root.something_at = this.created_at.ts_strftime("%Y-%b-%d %H:%M:%S") # In: {"created_at":"2020-08-14T11:50:26.371Z"} # Out: {"something_at":"2020-Aug-14 11:50:26"} ``` Format with microseconds using %f directive: ```bloblang root.something_at = this.created_at.ts_strftime("%Y-%b-%d %H:%M:%S.%f", "UTC") # In: {"created_at":"2020-08-14T11:50:26.371Z"} # Out: {"something_at":"2020-Aug-14 11:50:26.371000"} ``` ### [](#ts_strptime)ts\_strptime Parses a timestamp string using strptime format specifiers (like %Y, %m, %d) and outputs a timestamp object. Use ts\_parse for Go-style reference time formats instead. #### [](#parameters-109)Parameters | Name | Type | Description | | --- | --- | --- | | format | string | The format string using strptime specifiers (e.g., %Y-%m-%d). | #### [](#examples-158)Examples Parse date with abbreviated month using strptime format: ```bloblang root.doc.timestamp = this.doc.timestamp.ts_strptime("%Y-%b-%d") # In: {"doc":{"timestamp":"2020-Aug-14"}} # Out: {"doc":{"timestamp":"2020-08-14T00:00:00Z"}} ``` Parse datetime with microseconds using %f directive: ```bloblang root.doc.timestamp = this.doc.timestamp.ts_strptime("%Y-%b-%d %H:%M:%S.%f") # In: {"doc":{"timestamp":"2020-Aug-14 11:50:26.371000"}} # Out: {"doc":{"timestamp":"2020-08-14T11:50:26.371Z"}} ``` ### [](#ts_sub)ts\_sub Calculates the duration in nanoseconds between two timestamps (t1 - t2). Returns a signed integer: positive if t1 is after t2, negative if t1 is before t2. Use .abs() for absolute duration. #### [](#parameters-110)Parameters | Name | Type | Description | | --- | --- | --- | | t2 | timestamp | The timestamp to subtract from the target timestamp. | #### [](#examples-159)Examples Calculate absolute duration between two timestamps: ```bloblang root.between = this.started_at.ts_sub("2020-08-14T05:54:23Z").abs() # In: {"started_at":"2020-08-13T05:54:23Z"} # Out: {"between":86400000000000} ``` Calculate signed duration (can be negative): ```bloblang root.duration_ns = this.end_time.ts_sub(this.start_time) # In: {"start_time":"2020-08-14T10:00:00Z","end_time":"2020-08-14T11:30:00Z"} # Out: {"duration_ns":5400000000000} ``` ### [](#ts_sub_iso8601)ts\_sub\_iso8601 Subtracts an ISO 8601 duration from a timestamp with calendar-aware precision for years, months, and days. Useful when you need to subtract durations that account for variable month lengths or leap years. #### [](#parameters-111)Parameters | Name | Type | Description | | --- | --- | --- | | duration | string | Duration in ISO 8601 format (e.g., "P1Y2M3D" for 1 year, 2 months, 3 days) | #### [](#examples-160)Examples Subtract one year from a timestamp: ```bloblang root.last_year = this.created_at.ts_sub_iso8601("P1Y") # In: {"created_at":"2020-08-14T05:54:23Z"} # Out: {"last_year":"2019-08-14T05:54:23Z"} ``` Subtract a complex duration with multiple units: ```bloblang root.past_date = this.created_at.ts_sub_iso8601("P1Y2M3DT4H5M6S") # In: {"created_at":"2021-03-04T04:05:06Z"} # Out: {"past_date":"2020-01-01T00:00:00Z"} ``` ### [](#ts_tz)ts\_tz Converts a timestamp to a different timezone while preserving the moment in time. Accepts unix timestamps (seconds with optional decimal precision) or RFC 3339 formatted strings. #### [](#parameters-112)Parameters | Name | Type | Description | | --- | --- | --- | | tz | string | The timezone to change to. Use "UTC" for UTC, "Local" for local timezone, or an IANA Time Zone database location name like "America/New_York". | #### [](#examples-161)Examples Convert timestamp to UTC timezone: ```bloblang root.created_at_utc = this.created_at.ts_tz("UTC") # In: {"created_at":"2021-02-03T17:05:06+01:00"} # Out: {"created_at_utc":"2021-02-03T16:05:06Z"} ``` Convert timestamp to a specific timezone: ```bloblang root.created_at_ny = this.created_at.ts_tz("America/New_York") # In: {"created_at":"2021-02-03T16:05:06Z"} # Out: {"created_at_ny":"2021-02-03T11:05:06-05:00"} ``` ### [](#ts_unix)ts\_unix Converts a timestamp to a unix timestamp (seconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing seconds. #### [](#examples-162)Examples Convert RFC 3339 timestamp to unix seconds: ```bloblang root.created_at_unix = this.created_at.ts_unix() # In: {"created_at":"2009-11-10T23:00:00Z"} # Out: {"created_at_unix":1257894000} ``` Unix timestamp passthrough returns same value: ```bloblang root.timestamp = this.ts.ts_unix() # In: {"ts":1257894000} # Out: {"timestamp":1257894000} ``` ### [](#ts_unix_micro)ts\_unix\_micro Converts a timestamp to a unix timestamp with microsecond precision (microseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing microseconds. #### [](#examples-163)Examples Convert timestamp to microseconds since epoch: ```bloblang root.created_at_unix = this.created_at.ts_unix_micro() # In: {"created_at":"2009-11-10T23:00:00Z"} # Out: {"created_at_unix":1257894000000000} ``` Preserve microsecond precision from timestamp: ```bloblang root.precise_time = this.timestamp.ts_unix_micro() # In: {"timestamp":"2020-08-14T11:45:26.123456Z"} # Out: {"precise_time":1597405526123456} ``` ### [](#ts_unix_milli)ts\_unix\_milli Converts a timestamp to a unix timestamp with millisecond precision (milliseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing milliseconds. #### [](#examples-164)Examples Convert timestamp to milliseconds since epoch: ```bloblang root.created_at_unix = this.created_at.ts_unix_milli() # In: {"created_at":"2009-11-10T23:00:00Z"} # Out: {"created_at_unix":1257894000000} ``` Useful for JavaScript timestamp compatibility: ```bloblang root.js_timestamp = this.event_time.ts_unix_milli() # In: {"event_time":"2020-08-14T11:45:26.123Z"} # Out: {"js_timestamp":1597405526123} ``` ### [](#ts_unix_nano)ts\_unix\_nano Converts a timestamp to a unix timestamp with nanosecond precision (nanoseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing nanoseconds. #### [](#examples-165)Examples Convert timestamp to nanoseconds since epoch: ```bloblang root.created_at_unix = this.created_at.ts_unix_nano() # In: {"created_at":"2009-11-10T23:00:00Z"} # Out: {"created_at_unix":1257894000000000000} ``` Preserve full nanosecond precision: ```bloblang root.precise_time = this.timestamp.ts_unix_nano() # In: {"timestamp":"2020-08-14T11:45:26.123456789Z"} # Out: {"precise_time":1597405526123456789} ``` ## [](#type-coercion)Type coercion ### [](#array)array Converts a value to an array. #### [](#examples-166)Examples ```bloblang root.my_array = this.name.array() # In: {"name":"foobar bazson"} # Out: {"my_array":["foobar bazson"]} ``` ### [](#bool)bool Converts a value to a boolean with optional fallback. #### [](#parameters-113)Parameters | Name | Type | Description | | --- | --- | --- | | default (optional) | bool | An optional value to yield if the target cannot be parsed as a boolean. | #### [](#examples-167)Examples ```bloblang root.foo = this.thing.bool() root.bar = this.thing.bool(true) ``` ### [](#bytes)bytes Marshals a value into a byte array. #### [](#examples-168)Examples ```bloblang root.first_byte = this.name.bytes().index(0) # In: {"name":"foobar bazson"} # Out: {"first_byte":102} ``` ### [](#not_empty)not\_empty Ensures a value is not empty. #### [](#examples-169)Examples ```bloblang root.a = this.a.not_empty() # In: {"a":"foo"} # Out: {"a":"foo"} # In: {"a":""} # Out: Error("failed assignment (line 1): field `this.a`: string value is empty") # In: {"a":["foo","bar"]} # Out: {"a":["foo","bar"]} # In: {"a":[]} # Out: Error("failed assignment (line 1): field `this.a`: array value is empty") # In: {"a":{"b":"foo","c":"bar"}} # Out: {"a":{"b":"foo","c":"bar"}} # In: {"a":{}} # Out: Error("failed assignment (line 1): field `this.a`: object value is empty") ``` ### [](#not_null)not\_null Ensures a value is not null. #### [](#examples-170)Examples ```bloblang root.a = this.a.not_null() # In: {"a":"foobar","b":"barbaz"} # Out: {"a":"foobar"} # In: {"b":"barbaz"} # Out: Error("failed assignment (line 1): field `this.a`: value is null") ``` ### [](#number)number Converts a value to a number with optional fallback. #### [](#parameters-114)Parameters | Name | Type | Description | | --- | --- | --- | | default (optional) | float | An optional value to yield if the target cannot be parsed as a number. | #### [](#examples-171)Examples ```bloblang root.foo = this.thing.number() + 10 root.bar = this.thing.number(5) * 10 ``` ### [](#string)string Converts a value to a string representation. #### [](#examples-172)Examples ```bloblang root.nested_json = this.string() # In: {"foo":"bar"} # Out: {"nested_json":"{\"foo\":\"bar\"}"} ``` ```bloblang root.id = this.id.string() # In: {"id":228930314431312345} # Out: {"id":"228930314431312345"} ``` ### [](#timestamp)timestamp Converts a value to a timestamp with optional fallback. #### [](#parameters-115)Parameters | Name | Type | Description | | --- | --- | --- | | default (optional) | timestamp | An optional value to yield if the target cannot be parsed as a timestamp. | #### [](#examples-173)Examples ```bloblang root.foo = this.ts.timestamp() root.bar = this.none.timestamp(1234567890.timestamp()) ``` ### [](#type)type Returns the type of a value as a string. #### [](#examples-174)Examples ```bloblang root.bar_type = this.bar.type() root.foo_type = this.foo.type() # In: {"bar":10,"foo":"is a string"} # Out: {"bar_type":"number","foo_type":"string"} ``` ```bloblang root.type = this.type() # In: "foobar" # Out: {"type":"string"} # In: 666 # Out: {"type":"number"} # In: false # Out: {"type":"bool"} # In: ["foo", "bar"] # Out: {"type":"array"} # In: {"foo": "bar"} # Out: {"type":"object"} # In: null # Out: {"type":"null"} ``` ```bloblang root.type = content().type() # In: foobar # Out: {"type":"bytes"} ``` ```bloblang root.type = this.ts_parse("2006-01-02").type() # In: "2022-06-06" # Out: {"type":"timestamp"} ``` ## [](#deprecated)Deprecated ### [](#format_timestamp)format\_timestamp > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Formats a timestamp as a string using Go’s reference time format. Defaults to RFC 3339 if no format specified. The format uses "Mon Jan 2 15:04:05 -0700 MST 2006" as a reference. Accepts unix timestamps (with decimal precision) or RFC 3339 strings. Use ts\_strftime for strftime-style formats. #### [](#parameters-116)Parameters | Name | Type | Description | | --- | --- | --- | | format | string | The output format using Go’s reference time. | | tz (optional) | string | Optional timezone (e.g., 'UTC', 'America/New_York'). Defaults to input timezone or local time for unix timestamps. | ### [](#format_timestamp_strftime)format\_timestamp\_strftime > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Formats a timestamp as a string using strptime format specifiers (like %Y, %m, %d). Accepts unix timestamps (with decimal precision) or RFC 3339 strings. Supports %f for microseconds. Use ts\_format for Go-style reference time formats. #### [](#parameters-117)Parameters | Name | Type | Description | | --- | --- | --- | | format | string | The output format using strptime specifiers. | | tz (optional) | string | Optional timezone. Defaults to input timezone or local time for unix timestamps. | ### [](#format_timestamp_unix)format\_timestamp\_unix > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Converts a timestamp to a unix timestamp (seconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing seconds. ### [](#format_timestamp_unix_micro)format\_timestamp\_unix\_micro > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Converts a timestamp to a unix timestamp with microsecond precision (microseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing microseconds. ### [](#format_timestamp_unix_milli)format\_timestamp\_unix\_milli > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Converts a timestamp to a unix timestamp with millisecond precision (milliseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing milliseconds. ### [](#format_timestamp_unix_nano)format\_timestamp\_unix\_nano > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Converts a timestamp to a unix timestamp with nanosecond precision (nanoseconds since epoch). Accepts unix timestamps or RFC 3339 strings. Returns an integer representing nanoseconds. ### [](#parse_timestamp)parse\_timestamp > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Parses a timestamp string using Go’s reference time format and outputs a timestamp object. The format uses "Mon Jan 2 15:04:05 -0700 MST 2006" as a reference - show how this reference time would appear in your format. Use ts\_strptime for strftime-style formats instead. #### [](#parameters-118)Parameters | Name | Type | Description | | --- | --- | --- | | format | string | The format of the input string using Go’s reference time. | ### [](#parse_timestamp_strptime)parse\_timestamp\_strptime > ⚠️ **WARNING** > > This method is deprecated and will be removed in a future version. Parses a timestamp string using strptime format specifiers (like %Y, %m, %d) and outputs a timestamp object. Use ts\_parse for Go-style reference time formats instead. #### [](#parameters-119)Parameters | Name | Type | Description | | --- | --- | --- | | format | string | The format string using strptime specifiers (e.g., %Y-%m-%d). | --- # Page 370: Bloblang Playground **URL**: https://docs.redpanda.com/redpanda-connect/guides/bloblang/playground.md --- # Bloblang Playground --- title: Bloblang Playground latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: bloblang/playground page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: bloblang/playground.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/bloblang/playground.adoc description: Experiment with Bloblang in this interactive playground. Test mappings, process JSON inputs, and explore examples of real-time data transformations. page-git-created-date: "2024-12-03" page-git-modified-date: "2024-12-03" --- Enter your data in **Input**, write a Bloblang mapping to transform it, and see the result in **Output**. Or, [learn Bloblang](https://docs.redpanda.com/redpanda-connect/guides/bloblang/about/). × ### Keyboard Shortcuts × - Ctrl/Cmd + Enter Execute mapping - Ctrl/Cmd + Shift + X Clear all fields - Ctrl/Cmd + Shift + F Format JSON input - Ctrl/Cmd + S Copy share link - Ctrl/Cmd + E Export configuration - Ctrl/Cmd + / Toggle this help #### Claude AI Assistant Use Claude Code to help with Bloblang transformations - **/rpcn:blobl** - Generate mappings from natural language - **/rpcn:search** - Discover Redpanda Connect components - **/rpcn:pipeline** - Build complete streaming pipelines [Learn more about Claude Code for Redpanda →](https://github.com/redpanda-data/connect/blob/main/.claude-plugin/README.md) Access array items Add metadata Add new field Aggregation Array processing Conditional mapping Hello world Merge objects Nested object Remove sensitive data String manipulations Share Export Import ? Run Auto-run ⓘ #### Input ⓘ Auto-detect JSON Text CSV XML Ask AI Format Copy Clear #### Mapping ⓘ Ask AI Copy Clear ↓ #### Output ⓘ Copy Output will appear here... ▶ Advanced Options Optional #### Input Metadata Format Copy Clear #### Output Metadata Metadata will appear here... --- # Page 371: Bloblang Walkthrough **URL**: https://docs.redpanda.com/redpanda-connect/guides/bloblang/walkthrough.md --- # Bloblang Walkthrough --- title: Bloblang Walkthrough latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: bloblang/walkthrough page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: bloblang/walkthrough.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/bloblang/walkthrough.adoc description: A step by step introduction to Bloblang page-git-created-date: "2024-05-24" page-git-modified-date: "2026-03-31" --- Bloblang is the most advanced mapping language that you’ll learn from this walkthrough (probably). It is designed for readability, the power to shape even the most outrageous input documents, and to easily make erratic schemas bend to your will. Bloblang is the native mapping language of Redpanda Connect, but it has been designed as a general purpose technology ready to be adopted by other tools. In this walkthrough you’ll learn how to make new friends by mapping their documents, and lose old friends as they grow jealous and bitter of your mapping abilities. There are a few ways to execute Bloblang but the way we’ll do it in this guide is to pull a Redpanda Connect docker image and run the command `rpk connect blobl server`, which opens up an interactive Bloblang editor: ```sh docker pull docker.redpanda.com/redpandadata/connect:latest docker run -p 4195:4195 --rm docker.redpanda.com/redpandadata/connect blobl server --no-open --host 0.0.0.0 ``` > 📝 **NOTE: Alternatives** > > Alternatives > > For alternative Redpanda Connect installation options check out the [getting started guide](../../../get-started/quickstarts/rpk/). Next, open your browser at `http://localhost:4195` and you should see an app with three panels, the top-left is where you paste an input document, the bottom is your Bloblang mapping and on the top-right is the output. ## [](#your-first-assignment)Your first assignment The primary goal of a Bloblang mapping is to construct a brand new document by using an input document as a reference, which we achieve through a series of assignments. Bloblang is traditionally used to map JSON documents and that’s mostly what we’ll be doing in this walkthrough. The first mapping you’ll see when you open the editor is a single assignment: ```bloblang root = this # In: {"message":"hello world"} # Out: {"message":"hello world"} ``` On the left-hand side of the assignment is our assignment target, where `root` is a keyword referring to the root of the new document being constructed. On the right-hand side is a query which determines the value to be assigned, where `this` is a keyword that refers to the context of the mapping which begins as the root of the input document. As you can see the input document in the editor begins as a JSON object `{"message":"hello world"}`, and the output panel should show the result as: ```json { "message": "hello world" } ``` This output is a (neatly formatted) replica of the input document. This is the result of our mapping because we assigned the entire input document to the root of our new thing. Let’s create a brand new document by assigning a fresh object to the root: ```bloblang root = {} root.foo = this.message # In: {"message":"hello world"} # Out: {"foo":"hello world"} ``` Bloblang supports a bunch of [literal types](../about/#literals), and the first line of this mapping assigns an empty object literal to the root. The second line then creates a new field `foo` on that object by assigning it the value of `message` from the input document. You should see that our output has changed to: ```json { "foo": "hello world" } ``` In Bloblang, when the path that we assign to contains fields that are themselves unset then they are created as empty objects. This rule also applies to `root` itself, which means the mapping: ```bloblang root.foo.bar = this.message root.foo."buz me".baz = "I like mapping" # In: {"message":"hello world"} # Out: {"foo":{"bar":"hello world","buz me":{"baz":"I like mapping"}}} ``` Will automatically create the objects required to produce the output document: ```json { "foo": { "bar": "hello world", "buz me": { "baz": "I like mapping" } } } ``` Also note that we can use quotes in order to express path segments that contain symbols or whitespace. Great, let’s move on quick before our self-satisfaction gets in the way of progress. ## [](#basic-methods-and-functions)Basic methods and functions Nothing is ever good enough for you, why should the input document be any different? Usually in our mappings it’s necessary to mutate values whilst we map them over, this is almost always done with methods, of which [there are many](../methods/). To demonstrate we’re going to change our mapping to [uppercase](../methods/#uppercase) the field `message` from our input document: ```bloblang root.foo.bar = this.message.uppercase() root.foo."buz me".baz = "I like mapping" # In: {"message":"hello world"} # Out: {"foo":{"bar":"HELLO WORLD","buz me":{"baz":"I like mapping"}}} ``` As you can see the syntax for a method is similar to many languages, simply add a dot on the target value followed by the method name and arguments within brackets. With this method added our output document should look like this: ```json { "foo": { "bar": "HELLO WORLD", "buz me": { "baz": "I like mapping" } } } ``` Since the result of any Bloblang query is a value you can use methods on anything, including other methods. For example, we could expand our mapping of `message` to also replace `WORLD` with `EARTH` using the [`replace_all` method](../methods/#replace_all): ```bloblang root.foo.bar = this.message.uppercase().replace_all("WORLD", "EARTH") root.foo."buz me".baz = "I like mapping" # In: {"message":"hello world"} # Out: {"foo":{"bar":"HELLO EARTH","buz me":{"baz":"I like mapping"}}} ``` As you can see this method required some arguments. Methods support both nameless (like above) and named arguments, which are often literal values but can also be queries themselves. For example try out the following mapping using both named style and a dynamic argument: ```bloblang root.foo.bar = this.message.uppercase().replace_all(old: "WORLD", new: this.message.capitalize()) root.foo."buz me".baz = "I like mapping" # In: {"message":"hello world"} # Out: {"foo":{"bar":"HELLO Hello World","buz me":{"baz":"I like mapping"}}} ``` Woah, I think that’s the plot to Inception, let’s move onto functions. Functions are just boring methods that don’t have a target, and there are [plenty of them as well](../functions/). Functions are often used to extract information unrelated to the input document, such as [environment variables](../functions/#env), or to generate data such as [timestamps](../functions/#now) or [UUIDs](../functions/#uuid_v4). Since we’re completionists let’s add one to our mapping: ```bloblang root.foo.bar = this.message.uppercase().replace_all("WORLD", "EARTH") root.foo."buz me".baz = "I like mapping" root.foo.id = uuid_v4() # In: {"message":"hello world"} ``` Now I can’t tell you what the output looks like since it will be different each time it’s mapped, how fun! ### [](#deletions)Deletions Everything in Bloblang is an expression to be assigned, including deletions, which is a [function `deleted()`](../functions/#deleted). To illustrate let’s create a field we want to delete by changing our input to the following: ```json { "name": "fooman barson", "age": 7, "opinions": ["trucks are cool","trains are cool","chores are bad"] } ``` If we wanted a full copy of this document without the field `name` then we can assign `deleted()` to it: ```bloblang root = this root.name = deleted() # In: {"name":"fooman barson","age":7,"opinions":["trucks are cool","trains are cool","chores are bad"]} # Out: {"age":7,"opinions":["trucks are cool","trains are cool","chores are bad"]} ``` And it won’t be included in the output: ```json { "age": 7, "opinions": [ "trucks are cool", "trains are cool", "chores are bad" ] } ``` An alternative way to delete fields is the [method `without`](../methods/#without), our above example could be rewritten as a single assignment `root = this.without("name")`. However, `deleted()` is generally more powerful and will come into play more later on. ## [](#variables)Variables Sometimes it’s necessary to capture a value for later, but we might not want it to be added to the resulting document. In Bloblang we can achieve this with variables which are created using the `let` keyword, and can be referenced within subsequent queries with a dollar sign prefix: ```bloblang let id = uuid_v4() root.id_sha1 = $id.hash("sha1").encode("hex") root.id_md5 = $id.hash("md5").encode("hex") # In: {} ``` Variables can be assigned any value type, including objects and arrays. ## [](#unstructured-and-binary-data)Unstructured and binary data So far in all of our examples both the input document and our newly mapped document are structured, but this does not need to be so. Try assigning some literal value types directly to the `root`, such as a string `root = "hello world"`, or a number `root = 5`. You should notice that when a value type is assigned to the root the output is the raw value, and therefore strings are not quoted. This is what makes it possible to output data of any format, including encrypted, encoded or otherwise binary data. Unstructured mapping is not limited to the output. Rather than referencing the input document with `this`, where it must be structured, it is possible to reference it as a binary string with the [function `content`](../functions/#content), try changing your mapping to: ```bloblang root = content().uppercase() # In: hello world # Out: HELLO WORLD ``` When you add content to the input panel, it should be the same in the output panel, but in all uppercase. ## [](#conditionals)Conditionals In order to play around with conditionals let’s set our input to something structured: ```json { "pet": { "type": "cat", "is_cute": true, "treats": 5, "toys": 3 } } ``` In Bloblang all conditionals are expressions, this is a core principal of Bloblang and will be important later on when we’re mapping deeply nested structures. ### [](#if-expression)If expression The simplest conditional is the `if` expression, where the boolean condition does not need to be in parentheses. Let’s create a map that modifies the number of treats our pet receives based on a field: ```bloblang root = this root.pet.treats = if this.pet.is_cute { this.pet.treats + 10 } # In: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} # Out: {"pet":{"type":"cat","is_cute":true,"treats":15,"toys":3}} ``` Try that mapping out and you should see the number of treats in the output increased to 15. Now try changing the input field `pet.is_cute` to `false` and the output treats count should go back to the original 5. When a conditional expression doesn’t have a branch to execute then the assignment is skipped entirely, which means when the pet is not cute the value of `pet.treats` is unchanged (and remains the value set in the `root = this` assignment). We can add an `else` block to our `if` expression to remove treats entirely when the pet is not cute: ```bloblang root = this root.pet.treats = if this.pet.is_cute { this.pet.treats + 10 } else { deleted() } # In: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} # Out: {"pet":{"type":"cat","is_cute":true,"treats":15,"toys":3}} ``` This is possible because field deletions are expressed as assigned values created with the `deleted()` function. ### [](#if-statement)If statement The `if` keyword can also be used as a statement in order to conditionally apply a series of mapping assignments, the previous example can be rewritten as: ```bloblang root = this if this.pet.is_cute { root.pet.treats = this.pet.treats + 10 } else { root.pet.treats = deleted() } # In: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} # Out: {"pet":{"type":"cat","is_cute":true,"treats":15,"toys":3}} ``` Converting this mapping to use a statement has resulted in a more verbose mapping as we had to specify `root.pet.treats` multiple times as an assignment target. However, using `if` as a statement can be beneficial when multiple assignments rely on the same logic: ```bloblang root = this if this.pet.is_cute { root.pet.treats = this.pet.treats + 10 root.pet.toys = this.pet.toys + 10 } # In: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} # Out: {"pet":{"type":"cat","is_cute":true,"treats":15,"toys":13}} ``` More treats _and_ more toys! Lucky Spot! ### [](#match-expression)Match expression Another conditional expression is `match` which allows you to list many branches consisting of a condition and a query to execute separated with `=>`, where the first condition to pass is the one that is executed: ```bloblang root = this root.pet.toys = match { this.pet.treats > 5 => this.pet.treats - 5, this.pet.type == "cat" => 3, this.pet.type == "dog" => this.pet.toys - 3, this.pet.type == "horse" => this.pet.toys + 10, _ => 0, } # In: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} # Out: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} ``` Try executing that mapping with different values for `pet.type` and `pet.treats`. Match expressions can also specify a new context for the keyword `this` which can help reduce some of the boilerplate in your boolean conditions. The following mapping is equivalent to the previous: ```bloblang root = this root.pet.toys = match this.pet { this.treats > 5 => this.treats - 5, this.type == "cat" => 3, this.type == "dog" => this.toys - 3, this.type == "horse" => this.toys + 10, _ => 0, } # In: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} # Out: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} ``` Your boolean conditions can also be expressed as value types, in which case the context being matched will be compared to the value: ```bloblang root = this root.pet.toys = match this.pet.type { "cat" => 3, "dog" => 5, "rabbit" => 8, "horse" => 20, _ => 0, } # In: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} # Out: {"pet":{"type":"cat","is_cute":true,"treats":5,"toys":3}} ``` ## [](#error-handling)Error handling Bloblang can simplify handling errors. First, let’s take a look at what happens when errors _aren’t_ handled, change your input to the following: ```json { "palace_guards": 10, "angry_peasants": "I couldn't be bothered to ask them" } ``` And change your mapping to something simple like a number comparison: ```bloblang root.in_trouble = this.angry_peasants > this.palace_guards # In: {"palace_guards":10,"angry_peasants":"I couldn't be bothered to ask them"} ``` Uh oh! It looks like our canvasser was too lazy and our `angry_peasants` count was incorrectly set for this document. You should see an error in the output window that mentions something like `cannot compare types string (from field this.angry_peasants) and number (from field this.palace_guards)`, which means the mapping was abandoned. So what if we want to try and map something, but don’t care if it fails? In this case if we are unable to compare our angry peasants with palace guards then I would still consider us in trouble just to be safe. For that we have a special [method `catch`](../methods/#catch), which if we add to any query allows us to specify an argument to be returned when an error occurs. Since methods can be added to any query we can surround our arithmetic with brackets and catch the whole thing: ```bloblang root.in_trouble = (this.angry_peasants > this.palace_guards).catch(true) # In: {"palace_guards":10,"angry_peasants":"I couldn't be bothered to ask them"} # Out: {"in_trouble":true} ``` Now instead of an error we should see an output with `in_trouble` set to `true`. Try changing to value of `angry_peasants` to a few different values, including some numbers. One of the powerful features of `catch` is that when it is added at the end of a series of expressions and methods it will capture errors at any part of the series, allowing you to capture errors at any granularity. For example, the mapping: ```bloblang root.abort_mission = if this.mission.type == "impossible" { !this.user.motives.contains("must clear name") } else { this.mission.difficulty > 10 }.catch(false) # In: {"mission":{"type":"impossible","difficulty":5},"user":{"motives":["must clear name"]}} # Out: {"abort_mission":false} ``` Will catch errors caused by: - `this.mission.type` not being a string - `this.user.motives` not being an array - `this.mission.difficulty` not being a number But will always return `false` if any of those errors occur. Try it out with this input and play around by breaking some of the fields: ```json { "mission": { "type": "impossible", "difficulty": 5 }, "user": { "motives": ["must clear name"] } } ``` Now try out this mapping: ```bloblang root.abort_mission = if (this.mission.type == "impossible").catch(true) { !this.user.motives.contains("must clear name").catch(false) } else { (this.mission.difficulty > 10).catch(true) } # In: {"mission":{"type":"impossible","difficulty":5},"user":{"motives":["must clear name"]}} # Out: {"abort_mission":false} ``` This version is more granular and will capture each of the errors individually, with each error given a unique `true` or `false` fallback. ## [](#validation)Validation Sometimes errors are what we want. Failing a mapping with an error allows us to handle the bad document in other ways, such as routing it to a dead-letter queue or filtering it entirely. You can read about common Redpanda Connect error handling patterns for bad data in the [error handling guide](../../../configuration/error_handling/), but the first step is to create the error. Luckily, Bloblang has a range of ways of creating errors under certain circumstances, which can be used in order to validate the data being mapped. There are [a few helper methods](../methods/#type-coercion) that make validating and coercing fields nice and easy, try this mapping out: ```bloblang root.foo = this.foo.number() root.bar = this.bar.not_null() root.baz = this.baz.not_empty() # In: {"foo":5,"bar":"hello world","baz":[1,2,3]} # Out: {"foo":5,"bar":"hello world","baz":[1,2,3]} ``` With some of these sample inputs: ```json {"foo":"nope","bar":"hello world","baz":[1,2,3]} {"foo":5,"baz":[1,2,3]} {"foo":10,"bar":"hello world","baz":[]} ``` However, these methods don’t cover all use cases. The general purpose error throwing technique is the [`throw` function](../functions/#throw), which takes an argument string that describes the error. When it’s called it will throw a mapping error that abandons the mapping. For example, we can check the type of a field with the [method `type`](../methods/#type), and then throw an error if it’s not the type we expected: ```bloblang root.foos = if this.user.foos.type() == "array" { this.user.foos } else { throw("foos must be an array, but it ain't, what gives?") } # In: {"user":{"foos":[1,2,3]}} ``` Try this mapping out with a few sample inputs: ```json {"user":{"foos":[1,2,3]}} {"user":{"foos":"1,2,3"}} ``` ## [](#context)Context In Bloblang, when we refer to the context we’re talking about the value returned with the keyword `this`. At the beginning of a mapping the context starts off as a reference to the root of a structured input document, which is why the mapping `root = this` will result in the same document coming out as you put in. However, in Bloblang there are mechanisms whereby the context might change, we’ve already seen how this can happen within a `match` expression. Another useful way to change the context is by adding a bracketed query expression as a method to a query, which looks like this: ```bloblang root = this.foo.bar.(this.baz + this.buz) # In: {"foo":{"bar":{"baz":1,"buz":2}}} # Out: 3 ``` Within the bracketed query expression the context becomes the result of the query that it’s a method of, so within the brackets in the above mapping the value of `this` points to the result of `this.foo.bar`, and the mapping is therefore equivalent to: ```bloblang root = this.foo.bar.baz + this.foo.bar.buz # In: {"foo":{"bar":{"baz":1,"buz":2}}} # Out: 3 ``` With this handy trick the `throw` mapping from the validation section above could be rewritten as: ```bloblang root.foos = this.user.foos.(if this.type() == "array" { this } else { throw("foos must be an array, but it ain't, what gives?") }) # In: {"user":{"foos":[1,2,3]}} # Out: {"foos":[1,2,3]} ``` ### [](#naming-the-context)Naming the context Shadowing the keyword `this` with new contexts can look confusing in your mappings, and it also limits you to only being able to reference one context at any given time. As an alternative, Bloblang supports context capture expressions that look similar to lambda functions from other languages, where you can name the new context with the syntax ` -> `, which looks like this: ```bloblang root = this.foo.bar.(thing -> thing.baz + thing.buz) # In: {"foo":{"bar":{"baz":1,"buz":2}}} # Out: 3 ``` Within the brackets we now have a new field `thing`, which returns the context that would have otherwise been captured as `this`. This also means the value returned from `this` hasn’t changed and will continue to return the root of the input document. ## [](#coalescing)Coalescing Being able to open up bracketed query expressions on fields leads us onto another cool trick in Bloblang referred to as coalescing. It’s very common in the world of document mapping that due to structural deviations a value that we wish to obtain could come from one of multiple possible paths. To illustrate this problem change the input document to the following: ```json { "thing": { "article": { "id": "foo", "contents": "Some people did some stuff" } } } ``` Let’s say we wish to flatten this structure with the following mapping: ```bloblang root.contents = this.thing.article.contents # In: {"thing":{"article":{"id":"foo","contents":"Some people did some stuff"}}} # Out: {"contents":"Some people did some stuff"} ``` But articles are only one of many document types we expect to receive, where the field `contents` remains the same but the field `article` could instead be `comment` or `share`. In this case we could expand our map of `contents` to use a `match` expression where we check for the existence of `article`, `comment`, etc in the input document. However, a much cleaner way of approaching this is with the pipe operator (`|`), which in Bloblang can be used to join multiple queries, where the first to yield a non-null result is selected. Change your mapping to the following: ```bloblang root.contents = this.thing.article.contents | this.thing.comment.contents # In: {"thing":{"article":{"id":"foo","contents":"Some people did some stuff"}}} # Out: {"contents":"Some people did some stuff"} ``` And now try changing the field `article` in your input document to `comment`. You should see that the value of `contents` remains as `Some people did some stuff` in the output document. Now, rather than write out the full path prefix `this.thing` each time we can use a bracketed query expression to change the context, giving us more space for adding other fields: ```bloblang root.contents = this.thing.(this.article | this.comment | this.share).contents # In: {"thing":{"article":{"id":"foo","contents":"Some people did some stuff"}}} # Out: {"contents":"Some people did some stuff"} ``` And by the way, the keyword `this` within queries can be omitted and made implicit, which allows us to reduce this even further: ```bloblang root.contents = this.thing.(article | comment | share).contents # In: {"thing":{"article":{"id":"foo","contents":"Some people did some stuff"}}} # Out: {"contents":"Some people did some stuff"} ``` Finally, we can also add a pipe operator at the end to fallback to a literal value when none of our candidates exists: ```bloblang root.contents = this.thing.(article | comment | share).contents | "nothing" # In: {"thing":{"article":{"id":"foo","contents":"Some people did some stuff"}}} # Out: {"contents":"Some people did some stuff"} ``` Neat. ## [](#advanced-methods)Advanced methods What happens when you need to map all of the elements of an array? Or filter the keys of an object by their values? What if the fellowship just used the eagles to fly to mount doom? Bloblang offers a bunch of advanced methods for [manipulating structured data types](../methods/#object—​array-manipulation), let’s take a quick tour of some of the cooler ones. Set your input document to this list of things: ```json { "num_friends": 5, "things": [ { "name": "yo-yo", "quantity": 10, "is_cool": true }, { "name": "dish soap", "quantity": 50, "is_cool": false }, { "name": "scooter", "quantity": 1, "is_cool": true }, { "name": "pirate hat", "quantity": 7, "is_cool": true } ] } ``` Let’s say we wanted to reduce the `things` in our input document to only those that are cool and where we have enough of them to share with our friends. We can do this with a [`filter` method](../methods/#filter): ```bloblang root = this.things.filter(thing -> thing.is_cool && thing.quantity > this.num_friends) # In: {"num_friends":5,"things":[{"name":"yo-yo","quantity":10,"is_cool":true},{"name":"dish soap","quantity":50,"is_cool":false},{"name":"scooter","quantity":1,"is_cool":true},{"name":"pirate hat","quantity":7,"is_cool":true}]} # Out: [{"name":"yo-yo","quantity":10,"is_cool":true},{"name":"pirate hat","quantity":7,"is_cool":true}] ``` Try running that mapping and you’ll see that the output is reduced. What is happening here is that the `filter` method takes an argument that is a query, and that query will be mapped for each individual element of the array (where the context is changed to the element itself). We have captured the context into a field `thing` which allows us to continue referencing the root of the input with `this`. The `filter` method requires the query parameter to resolve to a boolean `true` or `false`, and if it resolves to `true` the element will be present in the resulting array, otherwise it is removed. Being able to express a query argument to be applied to a range in this way is one of the more powerful features of Bloblang, and when mapping complex structured data these advanced methods will likely be a common tool that you’ll reach for. Another such method is [`map_each`](../methods/#map_each), which allows you to mutate each element of an array, or each value of an object. Change your input document to the following: ```json { "talking_heads": [ "1:E.T. is a bad film,Pokemon corrupted an entire generation", "2:Digimon ripped off Pokemon,Cats are boring", "3:I'm important", "4:Science is just made up,The Pokemon films are good,The weather is good" ] } ``` Here we have an array of talking heads, where each element is a string containing an identifer, a colon, and a comma separated list of their opinions. We wish to map each string into a structured object, which we can do with the following mapping: ```bloblang root = this.talking_heads.map_each(raw -> { "id": raw.split(":").index(0), "opinions": raw.split(":").index(1).split(",") }) # In: {"talking_heads":["1:E.T. is a bad film,Pokemon corrupted an entire generation","2:Digimon ripped off Pokemon,Cats are boring","3:I'm important","4:Science is just made up,The Pokemon films are good,The weather is good"]} # Out: [{"id":"1","opinions":["E.T. is a bad film","Pokemon corrupted an entire generation"]},{"id":"2","opinions":["Digimon ripped off Pokemon","Cats are boring"]},{"id":"3","opinions":["I'm important"]},{"id":"4","opinions":["Science is just made up","The Pokemon films are good","The weather is good"]}] ``` The argument to `map_each` is a query where the context is the element, which we capture into the field `raw`. The result of the query argument will become the value of the element in the resulting array, and in this case we return an object literal. In order to separate the identifier from opinions we perform a `split` by colon on the raw string element and get the first substring with the `index` method. We then do the split again and extract the remainder, and split that by comma in order to extract all of the opinions to an array field. However, one problem with this mapping is that the split by colon is written out twice and executed twice. A more efficient way of performing the same thing is with the bracketed query expressions we’ve played with before: ```bloblang root = this.talking_heads.map_each(raw -> raw.split(":").(split_string -> { "id": split_string.index(0), "opinions": split_string.index(1).split(",") })) # In: {"talking_heads":["1:E.T. is a bad film,Pokemon corrupted an entire generation","2:Digimon ripped off Pokemon,Cats are boring","3:I'm important","4:Science is just made up,The Pokemon films are good,The weather is good"]} # Out: [{"id":"1","opinions":["E.T. is a bad film","Pokemon corrupted an entire generation"]},{"id":"2","opinions":["Digimon ripped off Pokemon","Cats are boring"]},{"id":"3","opinions":["I'm important"]},{"id":"4","opinions":["Science is just made up","The Pokemon films are good","The weather is good"]}] ``` > 📝 **NOTE: Challenge!** > > Challenge! > > Try updating that map so that only opinions that mention Pokemon are kept To find more methods for manipulating structured data types check out the [methods page](../methods/#object—​array-manipulation). ## [](#reusable-mappings)Reusable mappings Bloblang has cool methods, sure, but there’s nothing cooler than methods you’ve made yourself. When the going gets tough in the mapping world the best solution is often to create a named mapping, which you can do with the keyword `map`: ```bloblang map parse_talking_head { let split_string = this.split(":") root.id = $split_string.index(0) root.opinions = $split_string.index(1).split(",") } root = this.talking_heads.map_each(raw -> raw.apply("parse_talking_head")) # In: {"talking_heads":["1:E.T. is a bad film,Pokemon corrupted an entire generation","2:Digimon ripped off Pokemon,Cats are boring","3:I'm important","4:Science is just made up,The Pokemon films are good,The weather is good"]} # Out: [{"id":"1","opinions":["E.T. is a bad film","Pokemon corrupted an entire generation"]},{"id":"2","opinions":["Digimon ripped off Pokemon","Cats are boring"]},{"id":"3","opinions":["I'm important"]},{"id":"4","opinions":["Science is just made up","The Pokemon films are good","The weather is good"]}] ``` The body of a named map, encapsulated with squiggly brackets, is a totally isolated mapping where `root` now refers to a new value being created for each invocation of the map, and `this` refers to the root of the context provided to the map. Named maps are executed with the [method `apply`](../methods/#apply), which has a string parameter identifying the map to execute, this means it’s possible to dynamically select the target map. As you can see in the above example we were able to use a custom map in order to create our talking head objects without the object literal. Within a named map we can also create variables that exist only within the scope of the map. A nice feature of named mappings is that they can invoke themselves recursively, allowing you to define mappings that walk deeply nested structures. The following mapping will scrub all values from a document that contain the word "Voldemort" (case insensitive): ```bloblang map remove_naughty_man { root = match { this.type() == "object" => this.map_each(item -> item.value.apply("remove_naughty_man")), this.type() == "array" => this.map_each(ele -> ele.apply("remove_naughty_man")), this.type() == "string" => if this.lowercase().contains("voldemort") { deleted() }, this.type() == "bytes" => if this.lowercase().contains("voldemort") { deleted() }, _ => this, } } root = this.apply("remove_naughty_man") # In: {"summer_party":{"theme":"the woman in black","guests":["Emma Bunton","the seal I spotted in Trebarwith","Voldemort","The cast of Swiss Army Man","Richard"],"notes":{"lisa":"I don't think voldemort eats fish","monty":"Seals hate dance music"}},"crushes":["Richard is nice but he hates pokemon","Victoria Beckham but I think she's taken","Charlie but they're totally into Voldemort"]} ``` Try running that mapping with the following input document: ```json { "summer_party": { "theme": "the woman in black", "guests": [ "Emma Bunton", "the seal I spotted in Trebarwith", "Voldemort", "The cast of Swiss Army Man", "Richard" ], "notes": { "lisa": "I don't think voldemort eats fish", "monty": "Seals hate dance music" } }, "crushes": [ "Richard is nice but he hates pokemon", "Victoria Beckham but I think she's taken", "Charlie but they're totally into Voldemort" ] } ``` ## [](#unit-testing)Unit testing Redpanda Connect has it’s own [unit testing capabilities](../../../configuration/unit_testing/) that you can also use for your mappings. To start with save a mapping into a file called something like `naughty_man.blobl`, we can use the example above from the reusable mappings section: ```bloblang map remove_naughty_man { root = match { this.type() == "object" => this.map_each(item -> item.value.apply("remove_naughty_man")), this.type() == "array" => this.map_each(ele -> ele.apply("remove_naughty_man")), this.type() == "string" => if this.lowercase().contains("voldemort") { deleted() }, this.type() == "bytes" => if this.lowercase().contains("voldemort") { deleted() }, _ => this, } } root = this.apply("remove_naughty_man") ``` Next, we can define our unit tests in an accompanying YAML file in the same directory, let’s call this `naughty_man_test.yaml`: ```yaml tests: - name: test naughty man scrubber target_mapping: './naughty_man.blobl' environment: {} input_batch: - content: | { "summer_party": { "theme": "the woman in black", "guests": [ "Emma Bunton", "the seal I spotted in Trebarwith", "Voldemort", "The cast of Swiss Army Man", "Richard" ] } } output_batches: - - json_equals: { "summer_party": { "theme": "the woman in black", "guests": [ "Emma Bunton", "the dolphin I spotted in Trebarwith", "The cast of Swiss Army Man", "Richard" ] } } ``` As you can see we’ve defined a single test, where we point to our mapping file which will be executed in our test. We then specify an input message which is a reduced version of the document we tried out before, and finally we specify output predicates, which is a JSON comparison against the output document. We can execute these tests with `rpk connect test ./naughty_man_test.yaml`, Redpanda Connect will also automatically find our tests if you simply run `rpk connect test ./…​`. You should see an output something like: ```text Test 'naughty_man_test.yaml' failed Failures: --- naughty_man_test.yaml --- test naughty man scrubber [line 2]: batch 0 message 0: json_equals: JSON content mismatch { "summer_party": { "guests": [ "Emma Bunton", "the seal I spotted in Trebarwith" => "the dolphin I spotted in Trebarwith", "The cast of Swiss Army Man", "Richard" ], "theme": "the woman in black" } } ``` Because in actual fact our expected output is wrong, I’ll leave it to you to spot the error. Once the test is fixed you should see: ```text Test 'naughty_man_test.yaml' succeeded ``` And now our mapping, should we need to expand it in the future, is better protected against regressions. You can read more about the Redpanda Connect unit test specification, including alternative output predicates, in [this document](../../../configuration/unit_testing/). --- # Page 372: Cloud Credentials **URL**: https://docs.redpanda.com/redpanda-connect/guides/cloud.md --- # Cloud Credentials --- title: Cloud Credentials latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: cloud/index page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: cloud/index.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/cloud/index.adoc description: Explore how to deploy and manage Redpanda Connect in the cloud, including setup, configuration, and optimization tips. page-git-created-date: "2024-06-03" page-git-modified-date: "2024-06-04" --- - [Amazon Web Services](aws/) Find out about AWS components in Redpanda Connect. - [Authenticate to Amazon Aurora using IAM roles](aws-iam-aurora/) Configure IAM roles to authenticate Redpanda Connect pipelines to Amazon Aurora on AWS without static credentials. - [Google Cloud Platform](gcp/) Find out about GCP components in Redpanda Connect. --- # Page 373: Authenticate to Amazon Aurora using IAM roles **URL**: https://docs.redpanda.com/redpanda-connect/guides/cloud/aws-iam-aurora.md --- # Authenticate to Amazon Aurora using IAM roles --- title: Authenticate to Amazon Aurora using IAM roles latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: cloud/aws-iam-aurora page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: cloud/aws-iam-aurora.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/cloud/aws-iam-aurora.adoc description: Configure IAM roles to authenticate Redpanda Connect pipelines to Amazon Aurora on AWS without static credentials. page-git-created-date: "2026-03-17" page-git-modified-date: "2026-03-17" --- On AWS, you can use IAM Roles for Service Accounts (IRSA) to authenticate Redpanda Connect pipelines to Amazon Aurora without storing static credentials. ## [](#how-it-works)How it works Authentication happens through a two-hop role chain: 1. The Redpanda Connect pod uses a projected OIDC token to assume the cluster’s pipeline IAM role (`redpanda--redpanda-connect-pipeline`) using `sts:AssumeRoleWithWebIdentity` (IRSA). 2. The pipeline role assumes a database-specific role in your Aurora account using `sts:AssumeRole`. 3. The database role generates a short-lived RDS IAM authentication token that Aurora accepts as a password. Both roles must exist with trust relationships that permit this chain. The pipeline role is managed by Redpanda — you add a single inline policy to it. The database connect role is created and owned by you. ## [](#prerequisites)Prerequisites - Redpanda Connect deployed on AWS - An Aurora cluster (PostgreSQL or MySQL) - Permissions to create IAM roles and attach inline policies in both the Redpanda and Aurora AWS accounts - An IAM role associated with the Redpanda Connect pod (see [Step 1: Find the pipeline IAM role name](#step-1-find-the-pipeline-iam-role-name)) ## [](#step-1-find-the-pipeline-iam-role-name)Step 1: Find the pipeline IAM role name For Redpanda Cloud BYOC deployments, the pipeline role is pre-created in the AWS account where your cluster is deployed and follows this naming convention: redpanda--redpanda-connect-pipeline > ⚠️ **CAUTION** > > Do not modify the existing policies attached to this role. Redpanda manages these policies and reverts manual changes automatically. Only add new inline policies. For self-managed deployments, use the IAM role associated with your Redpanda Connect pod or EC2 instance. ## [](#step-2-find-the-aurora-cluster-resource-id)Step 2: Find the Aurora cluster resource ID The IAM policy for RDS IAM authentication requires the Aurora cluster resource ID (not the cluster identifier). Retrieve it with: ```bash aws rds describe-db-clusters \ --query "DBClusters[?DBClusterIdentifier==''].[DbClusterResourceId]" \ --output text \ --profile ``` The resource ID has the format `cluster-`. ## [](#step-3-create-the-database-connect-role-aurora-account)Step 3: Create the database connect role (Aurora account) In the AWS account where Aurora is hosted, create an IAM role with the following permission policy: permission-policy.json ```json { "Version": "2012-10-17", "Statement": [ { "Action": "rds-db:connect", "Effect": "Allow", "Resource": "arn:aws:rds-db:::dbuser:cluster-/" } ] } ``` Replace the following placeholders: - ``: The AWS region of the Aurora cluster - ``: The AWS account ID that hosts Aurora - ``: The cluster resource ID from [Step 2: Find the Aurora cluster resource ID](#step-2-find-the-aurora-cluster-resource-id) - ``: The database user configured for IAM authentication ### [](#trust-policy)Trust policy Attach the following trust policy to the role to allow the Redpanda pipeline role to assume it: trust-policy.json ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::role/" }, "Action": "sts:AssumeRole" } ] } ``` Replace the following placeholders: - ``: The AWS account ID where the Redpanda Connect pipeline role lives - ``: The pipeline role name from [Step 1: Find the pipeline IAM role name](#step-1-find-the-pipeline-iam-role-name) > 📝 **NOTE** > > The database connect role must be owned by the same AWS account as the Aurora instance. For cross-account setups, create this role in the Aurora account, not the Redpanda account. ### [](#required-tag)Required tag The database connect role must have the following tag. Without it, the pipeline role cannot assume the database role. | Key | Value | | --- | --- | | redpanda_scope_redpanda_connect | true | ### [](#create-the-role)Create the role Run the following AWS CLI commands to create the role and apply the permission policy and tag: ```bash aws iam create-role \ --role-name "" \ --assume-role-policy-document file://trust-policy.json \ --profile aws iam put-role-policy \ --role-name "" \ --policy-name "aurora-rds-connect" \ --policy-document file://permission-policy.json \ --profile aws iam tag-role \ --role-name "" \ --tags Key=redpanda_scope_redpanda_connect,Value=true \ --profile ``` ## [](#step-4-grant-the-pipeline-role-cross-account-access-redpanda-account)Step 4: Grant the pipeline role cross-account access (Redpanda account) In the AWS account where the Redpanda Connect pipeline role lives, add an inline policy to the pipeline role (identified in [Step 1: Find the pipeline IAM role name](#step-1-find-the-pipeline-iam-role-name)) to allow it to assume your database connect role: inline-policy.json ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "sts:AssumeRole", "Resource": "arn:aws:iam:::role/", "Condition": { "StringEquals": { "aws:ResourceTag/redpanda_scope_redpanda_connect": "true" } } } ] } ``` Attach the policy using the AWS CLI: ```bash aws iam put-role-policy \ --role-name "" \ --policy-name "allow-x-account-db-connect" \ --policy-document file://inline-policy.json \ --profile ``` To verify the policy is attached: ```bash aws iam get-role-policy \ --role-name "" \ --policy-name "allow-x-account-db-connect" \ --profile ``` ## [](#step-5-configure-the-network)Step 5: Configure the network The Aurora security group must allow inbound traffic from your Redpanda Connect cluster’s outbound IP addresses on the database port (5432 for PostgreSQL, 3306 for MySQL). For Redpanda Cloud BYOC deployments, retrieve the NAT Gateway IPs using the Redpanda Cloud API: ```bash AUTH_TOKEN=$(curl -s --request POST \ --url 'https://auth.prd.cloud.redpanda.com/oauth/token' \ --header 'content-type: application/x-www-form-urlencoded' \ --data grant_type=client_credentials \ --data client_id="${REDPANDA_CLIENT_ID}" \ --data client_secret="${REDPANDA_CLIENT_SECRET}" \ --data audience=cloudv2-production.redpanda.cloud | jq -r '.access_token') curl -s -X GET "https://api.cloud.redpanda.com/v1/clusters/${RP_CLUSTER_ID}" \ -H "Authorization: Bearer ${AUTH_TOKEN}" | jq .cluster.nat_gateways ``` ## [](#step-6-configure-the-pipeline)Step 6: Configure the pipeline In your Redpanda Connect pipeline YAML, set the `aws.roles` field to the ARN of the database connect role created in [Step 3: Create the database connect role (Aurora account)](#step-3-create-the-database-connect-role-aurora-account). This configuration applies to the `postgres_cdc`, `pg_stream`, and `mysql_cdc` inputs. For the `postgres_cdc` and `pg_stream` inputs: ```yaml input: postgres_cdc: dsn: "host= port=5432 user= dbname= sslmode=require" aws: enabled: true region: endpoint: roles: - role: arn:aws:iam:::role/ ``` For the `mysql_cdc` input: ```yaml input: mysql_cdc: dsn: "@tcp(:3306)/?tls=true" aws: enabled: true region: endpoint: roles: - role: arn:aws:iam:::role/ ``` The `aws.roles` field accepts an array of role ARNs, which are assumed in sequence. This supports chaining multiple role assumptions for more complex cross-account setups. ## [](#troubleshoot)Troubleshoot ### [](#is-not-authorized-to-perform-stsassumerole)is not authorized to perform: sts:AssumeRole The inline assume-role policy is missing from the Redpanda Connect pipeline role, or the tag condition is not met. 1. Verify that the database connect role has the `redpanda_scope_redpanda_connect: true` tag. 2. Attach the inline policy as described in [Step 4: Grant the pipeline role cross-account access (Redpanda account)](#step-4-grant-the-pipeline-role-cross-account-access-redpanda-account). The error clears automatically once the policy is in place. Monitor pipeline logs to confirm: ```bash rpk topic consume __redpanda.connect.logs --offset end | grep '' ``` Look for a message like `postgres_cdc input go active` or `mysql_cdc input go active` to confirm successful authentication. --- # Page 374: Amazon Web Services **URL**: https://docs.redpanda.com/redpanda-connect/guides/cloud/aws.md --- # Amazon Web Services --- title: Amazon Web Services latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: cloud/aws page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: cloud/aws.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/cloud/aws.adoc description: Find out about AWS components in Redpanda Connect. page-git-created-date: "2024-05-24" page-git-modified-date: "2026-03-17" --- There are many components within Redpanda Connect which utilize AWS services. You will find that each of these components contains a configuration section under the field `credentials`, of the format: ```yml credentials: profile: "" id: "" secret: "" token: "" role: "" role_external_id: "" ``` This section contains many fields and it isn’t immediately clear which of them are compulsory and which aren’t. This document aims to make it clear what each field is responsible for and how it might be used. ## [](#credentials)Credentials The first thing to make clear is that _all_ of these fields are optional. When all fields are left blank Redpanda Connect will attempt to load credentials from a shared credentials file (`~/.aws/credentials`). The profile loaded will be `default` unless the `AWS_PROFILE` environment variable is set. ## [](#explicit-credentials)Explicit credentials By explicitly setting the credentials you are using at the component level it’s possible to connect to components using different accounts within the same Redpanda Connect process. ### [](#select-a-profile)Select a profile If you are using your shared credentials file but wish to explicitly select a profile set the `profile` field: ```yml credentials: profile: foo ``` ### [](#manual)Manual If you are using long term credentials for your account you only need to set the fields `id` and `secret`: ```yml credentials: id: foo # aws_access_key_id secret: bar # aws_secret_access_key ``` If you are using short term credentials then you will also need to set the field `token`: ```yml credentials: id: foo # aws_access_key_id secret: bar # aws_secret_access_key token: baz # aws_session_token ``` ## [](#assume-a-role)Assume a role It’s also possible to configure Redpanda Connect to [assume a role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html) using your credentials by setting the field `role` to your target role ARN. ```yml credentials: role: fooarn # Role ARN ``` This does NOT require explicit credentials, but it’s possible to use both. If you need to assume a role owned by another organization they might require you to [provide an external ID](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html), in which case place it in the field `role_external_id`: ```yml credentials: role: fooarn # Role ARN role_external_id: bar_id ``` ## [](#related-guides)Related guides - [Authenticate to Amazon Aurora using IAM roles](../aws-iam-aurora/) --- # Page 375: Google Cloud Platform **URL**: https://docs.redpanda.com/redpanda-connect/guides/cloud/gcp.md --- # Google Cloud Platform --- title: Google Cloud Platform latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: cloud/gcp page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: cloud/gcp.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/cloud/gcp.adoc description: Find out about GCP components in Redpanda Connect. page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-09" --- There are many components within Redpanda Connect which utilize Google Cloud Platform (GCP) services. You will find that each of these components require valid credentials. When running Redpanda Connect inside a Google Cloud environment that has a [default service account](https://cloud.google.com/iam/docs/service-accounts#default), it can automatically retrieve the service account credentials to call Google Cloud APIs through a library called Application Default Credentials (ADC). Otherwise, if your application runs outside Google Cloud environments that provide a default service account, you need to manually create one. Once you have a service account set up which has the required permissions, you can [create](https://console.cloud.google.com/apis/credentials/serviceaccountkey) a new Service Account Key and download it as a JSON file. Then all you need to do set the path to this JSON file in the `GOOGLE_APPLICATION_CREDENTIALS` environment variable. Please refer to [this document](https://cloud.google.com/docs/authentication/production) for details. --- # Page 376: Message Delivery Semantics **URL**: https://docs.redpanda.com/redpanda-connect/guides/delivery_semantics.md --- # Message Delivery Semantics --- title: Message Delivery Semantics latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: delivery_semantics page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: delivery_semantics.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/delivery_semantics.adoc description: Learn about Redpanda Connect's transactional model and error handling page-git-created-date: "2025-07-22" page-git-modified-date: "2025-12-17" --- Redpanda Connect guarantees strong message delivery semantics across its data pipelines by leveraging a transactional model centered around message batch acknowledgment. This ensures reliable, scalable, and predictable data movement between input and output systems, with particular support for **at-least-once delivery** without requiring intermediate message persistence. This topic describes how Redpanda Connect manages message delivery across the Input → Pipeline → Output architecture. ![delivery semantics arch](../../shared/_images/delivery-semantics-arch.svg) ## [](#delivery-guarantees)Delivery guarantees An `ack()` function is associated with every batch of messages. This `ack()` function is a callback function provided by the input. It’s the output’s job to call the `ack()` function to inform the input of the success ("ack") or failure (negative ack or "nack") of the batch. The input can then decide what to do, for example, auto-retry nacks, propagate the nack upstream to input connectors that support it (Google PubSub), or drop the batch. When connecting to **at-least-once** inputs and outputs, Redpanda Connect guarantees **at-least-once** delivery without having to persist messages during transit. Key benefits: - Messages are retained unless configuration explicitly drops them - Every message is processed at-least-once, with optional retry logic for failures - Duplicates may occur during failure recovery, as is typical of at-least-once systems - Delivery status is communicated back to the input, enabling robust retry and propagation strategies - No need for disk-based checkpointing, reducing latency and infrastructure complexity ## [](#caveats-to-at-least-once)Caveats to at-least-once At-least-once delivery in a Redpanda Connect pipeline means every message is delivered at-least-once even if failures occur (which could result in some duplicates, but no lost data). Introducing a distributed or externally persisted state into a pipeline can compromise this guarantee. The pipeline’s built-in fault tolerance is based on the assumption that if an output fails, the message will be retried or reprocessed upon recovery. However, an external state (such as a remote cache or database used within the pipeline) may not reset on failure. It could retain an indication that a message was processed even if that message was never successfully processed by the output, and this mismatch can lead to message loss during recovery. For example, a pipeline uses a distributed cache to keep track of message IDs that have been seen in order to filter out duplicates. When a new message arrives, its ID is added to the cache immediately to mark it as processed. Now imagine that the pipeline’s output connector fails or the pipeline instance crashes just after updating the cache, but before the message actually reaches the destination. When the system comes back up, it will retry processing any unacknowledged messages, but because the cache still contains that message ID from the earlier attempt, the pipeline treats the message as a duplicate and drops it. As a result, that message is never delivered by the output at all, which is a direct violation of at-least-once guarantees. One way to mitigate this risk is to avoid using a state that outlives the pipeline process. For instance, keeping the cache in-memory (local to each pipeline instance) means it won’t preserve entries through a crash. If the process restarts, an in-memory cache would start fresh, allowing the system to reprocess any message that didn’t complete earlier instead of mistakenly skipping it. If you combine this with consistent partitioning of messages by key across horizontally scaled pipeline instances (so that each key is always handled by the same instance), you can still perform de-duplication within each partition while keeping state isolated per instance. In this setup, a failure that restarts one instance will clear that instance’s cache, and upon recovery it will reprocess its pending messages without interference from stale state. Even with such mitigations, if reliable delivery is paramount, a safer approach is to avoid relying on any external persistent state for eliminating duplicates within the pipeline. Instead, let the pipeline run with its native at-least-once behavior (allowing potential duplicates on retries) and design the downstream systems to handle duplicates idempotently. ## [](#error-handling)Error handling Redpanda Connect embraces a declarative and opt-in error-handling strategy: - A message error is represented as a flag, not a dropped message - If not explicitly handled in the pipeline, error messages flow through to the output stage - Error-handling components (for example, `try`, or `catch`) let you: - Retry failed operations - Route errors to a dead-letter queue - Fall back to alternate logic or output targets This model provides flexibility in building resilient pipelines without introducing unnecessary complexity. Refer to the specific [Error Handling](https://docs.redpanda.com/redpanda-connect/configuration/error_handling/) documentation for further details and examples. --- # Page 377: Migration **URL**: https://docs.redpanda.com/redpanda-connect/guides/migration.md --- # Migration --- title: Migration latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: migration/index page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: migration/index.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/migration/index.adoc description: Migrate between different Redpanda Connect versions, including preparation steps and best practices. page-git-created-date: "2024-06-03" page-git-modified-date: "2026-01-27" --- > ❗ **IMPORTANT** > > This section is related to migrating to a newer version of Redpanda Connect. For information about migrating your data using Redpanda Migrator, see [Redpanda Migrator](../../cookbooks/redpanda_migrator/). - [Migrate to the Unified Redpanda Migrator](migrate-unified-redpanda-migrator/) Learn how to migrate from legacy migrator components to the unified \`redpanda\_migrator\` input/output pair in Redpanda Connect 4.67.5+. - [Migrating to Version 4](v4/) Learn how to migrate to Redpanda Connect v4, with insights into new functionalities and migration tips. - [Migrating to Version 3](v3/) Learn how to migrate to Redpanda Connect v3. - [Migrating to Version 2](v2/) Learn how to migrate to Redpanda Connect v2. --- # Page 378: Migrate to the Unified Redpanda Migrator **URL**: https://docs.redpanda.com/redpanda-connect/guides/migration/migrate-unified-redpanda-migrator.md --- # Migrate to the Unified Redpanda Migrator --- title: Migrate to the Unified Redpanda Migrator latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: migration/migrate-unified-redpanda-migrator page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: migration/migrate-unified-redpanda-migrator.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/migration/migrate-unified-redpanda-migrator.adoc description: Learn how to migrate from legacy migrator components to the unified `redpanda_migrator` input/output pair in Redpanda Connect 4.67.5+. page-git-created-date: "2025-10-24" page-git-modified-date: "2026-03-31" --- > ❗ **IMPORTANT** > > This page is about migrating to a newer version of Redpanda Connect. For information about migrating your data using Redpanda Migrator, see [Redpanda Migrator](../../../cookbooks/redpanda_migrator/). This guide explains how to migrate from legacy migrator components (`redpanda_migrator_bundle`, `legacy_redpanda_migrator` and `legacy_redpanda_migrator_offsets`) to the unified `redpanda_migrator` input/output pair introduced in Redpanda Connect 4.67.5+. The unified migrator consolidates all migration logic into a single input/output pair, simplifying configuration and improving reliability. ## [](#overview)Overview | Available in | Redpanda Connect 4.67.5+ | | --- | --- | | Legacy status | Deprecated in 4.67.5, removed in 4.85.0 | | Compatibility | Not backward-compatible | | Configuration model | One input and one output, paired by label | | Primary control | All migration logic resides in the output component | Key concepts: - Components are paired by matching `label` values. - The input defines the source cluster and schema registry. - The output defines the destination cluster, schema registry, and migration behavior. - Topic mapping and consumer group migration are configured in the output. ## [](#architectural-changes)Architectural changes ### [](#legacy-architecture)Legacy architecture A complex bundle (`redpanda_migrator_bundle`) that managed three subcomponents: - `redpanda_migrator`: Data transfer - `schema_registry`: Schema synchronization - `redpanda_migrator_offsets`: Consumer group offsets This design required complex internal routing and sequencing. ### [](#unified-architecture)Unified architecture A single `redpanda_migrator` input/output pair replaces the bundle: - **Input**: Consumes from the source Kafka cluster. - **Output**: Handles topic creation, schema synchronization, ACLs, and consumer group offsets. Benefits: - Simplified setup: all configuration consolidated in one output component. - Improved coordination: no internal routing or wrapper logic. - Enhanced control: fine-grained schema and topic options, improved offset handling. ## [](#migration-steps)Migration steps Follow this checklist in order to ensure a safe, low-risk migration. - Back up your existing configurations. - Add new `input.redpanda_migrator` and `output.redpanda_migrator` components with matching labels. - Move source Kafka and Schema Registry settings to the input. - Move destination Kafka and Schema Registry settings to the output. - Replace `topic_prefix` with `topic` using interpolation syntax. - Move offset settings to `output.redpanda_migrator.consumer_groups`. - Remove deprecated fields. - Validate configuration with `rpk connect lint`. - Test using non-production topics first. - Monitor logs and performance during migration. - Remove legacy configuration after successful migration. ## [](#field-mapping-reference)Field mapping reference ### [](#bundle-wrapper-redpanda_migrator_bundle)Bundle wrapper (`redpanda_migrator_bundle`) #### [](#input-mapping)Input mapping | Legacy Field | New Location | Status | Notes | | --- | --- | --- | --- | | redpanda_migrator | input.redpanda_migrator | Moved | Source cluster connection | | schema_registry | input.redpanda_migrator.schema_registry | Moved | Source schema registry | | migrate_schemas_before_data | - | Removed | Controlled by output schema interval | | consumer_group_offsets_poll_interval | output.redpanda_migrator.consumer_groups.interval | Moved | Now controls sync frequency | #### [](#output-mapping)Output mapping | Legacy Field | New Location | Status | Notes | | --- | --- | --- | --- | | redpanda_migrator | output.redpanda_migrator | Moved | Destination cluster configuration | | schema_registry | output.redpanda_migrator.schema_registry | Moved | Destination schema registry | | translate_schema_ids | output.redpanda_migrator.schema_registry.translate_ids | Moved | Schema ID translation | | input_bundle_label | label | Replaced | Input and output paired by label | ### [](#data-migration-fields)Data migration fields | Legacy Field | New Location | Status | Notes | | --- | --- | --- | --- | | All (*) | input.redpanda_migrator.* | Moved | Direct mapping | | topics (explicit list) | input.redpanda_migrator.topics | Unchanged | Still supported for explicit lists | | regexp_topics: true | input.redpanda_migrator.regexp_topics_include, regexp_topics_exclude | Deprecated | Use include/exclude arrays for pattern-based selection | | topic_prefix | output.redpanda_migrator.topic | Replaced | Use interpolation, for example 'prefix_${! @kafka_topic }' | | replication_factor_override, replication_factor | output.redpanda_migrator.topic_replication_factor | Replaced | Unified field | | input_resource | label | Replaced | Label pairing replaces internal routing | | - | output.redpanda_migrator.provenance_header | New | Optional header for tracking message source cluster | ### [](#schema-migration-fields)Schema migration fields | Legacy Field | New Location | Status | Notes | | --- | --- | --- | --- | | Connection fields | input.redpanda_migrator.schema_registry.* | Moved | Source schema registry | | subject_filter | output.redpanda_migrator.schema_registry.include, exclude | Replaced | Use regex lists for filtering | | include_deleted | output.redpanda_migrator.schema_registry.include_deleted | Moved | Configured on destination | | backfill_dependencies | output.redpanda_migrator.schema_registry.versions | Replaced | Choose all or latest | ### [](#consumer-group-offset-migration)Consumer group offset migration The `redpanda_migrator_offsets` pair is replaced by the `consumer_groups` block in the output. | Legacy Component | New Location | Status | Notes | | --- | --- | --- | --- | | redpanda_migrator_offsets (input/output) | output.redpanda_migrator.consumer_groups | Replaced | Unified control block | ## [](#migration-example)Migration example The following example demonstrates a complete migration from legacy to unified components. Legacy configuration ```yaml input: label: "source_cluster" redpanda_migrator_bundle: legacy_redpanda_migrator: seed_brokers: [ "source-kafka:9092" ] topics: [ "orders", "payments" ] consumer_group: "migration_group" schema_registry: url: "http://source-registry:8081" migrate_schemas_before_data: false consumer_group_offsets_poll_interval: 30s output: redpanda_migrator_bundle: legacy_redpanda_migrator: seed_brokers: [ "destination-redpanda:9092" ] topic_prefix: "migrated_" schema_registry: url: "http://destination-registry:8081" translate_schema_ids: true input_bundle_label: "source_cluster" ``` Unified configuration ```yaml input: label: "migration_pipeline" (1) redpanda_migrator: # Source Kafka settings seed_brokers: [ "source-kafka:9092" ] # Pattern-based topic selection (for migrating all topics except system topics) # Note: You can still use explicit lists: topics: [ "orders", "payments" ] regexp_topics_include: [ '.' ] (2) regexp_topics_exclude: [ '^_' ] (3) consumer_group: "migration_group" # Source Schema Registry settings schema_registry: url: "http://source-registry:8081" output: label: "migration_pipeline" (4) redpanda_migrator: # Destination Redpanda settings seed_brokers: [ "destination-redpanda:9092" ] # Topic mapping (replaces topic_prefix) topic: 'migrated_${! @kafka_topic }' (5) # Add source cluster tracking header provenance_header: "x-source-cluster" (6) # Destination Schema Registry and migration settings schema_registry: url: "http://destination-registry:8081" translate_ids: true # Rename subjects subject: 'migrated_${! metadata("schema_registry_subject") }' # Consumer group migration settings consumer_groups: enabled: true interval: 30s (7) ``` | 1 | Labels are now used for pairing input and output. | | --- | --- | | 2 | Match all topics using regex pattern. | | 3 | Exclude internal/system topics starting with underscore. | | 4 | Matching label pairs the input and output components. | | 5 | Use interpolation syntax to replicate topic_prefix behavior. | | 6 | Adds a header to track which cluster messages originated from, useful for debugging and auditing. | | 7 | Replaces consumer_group_offsets_poll_interval. | ## [](#validation)Validation Before running, validate your configuration: ```bash rpk connect lint config.yaml ``` Then test on a small set of topics before running full migrations. ## [](#troubleshooting)Troubleshooting | Problem | Likely Cause | Solution | | --- | --- | --- | | Labels do not match | Input and output labels differ | Use identical, case-sensitive labels. | | Topic interpolation errors | Incorrect syntax | Use topic: 'prefix_${! @kafka_topic }' with quotes and !. | | Schema registry connection fails | Incorrect registry placement | The source registry must be in the input. The destination registry must be in the output. | | Consumer group migration not working | Missing consumer_groups.enabled: true | Ensure consumer group migration is explicitly enabled. | ## [](#after-migration)After migration After verifying that the new migrator works as expected: - Remove legacy configuration files. - Update internal documentation and runbooks. - Train your team on the new configuration model. - See the [`redpanda_migrator` output](../../../components/outputs/redpanda_migrator/) reference for advanced configuration options. --- # Page 379: Migrating to Version 2 **URL**: https://docs.redpanda.com/redpanda-connect/guides/migration/v2.md --- # Migrating to Version 2 --- title: Migrating to Version 2 latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: migration/v2 page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: migration/v2.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/migration/v2.adoc description: Learn how to migrate to Redpanda Connect v2. page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- Redpanda Connect version 2 comes with a small number of backwards incompatible changes that are organized into three sections; [configuration](#configuration), [service](#service) and [Go API](#go-api). These changes are considered minor and therefore upgrading is not expected to cause significant problems for any users. ## [](#configuration)Configuration ### [](#type-inference)Type inference Version 2 comes with the ability to infer the `type` of components in configuration files whenever the field is omitted. You can read more about this behavior [here](../../../configuration/about/#concise-configuration). This feature is not expected to impact the vast majority of users. However, there is one exception where a malformed section containing unused type parameters but a missing `type` field will be interpreted differently. For example, the following config: ```yml pipeline: processors: - # type: text text: operator: set value: "delete all your content" ``` In V1 would be interpreted as a `bounds_check` processor as it is the default processor type, whereas V2 would infer this to be a `text` processor based on its fields. #### [](#migration-guide)Migration guide Most users should not be impacted by this change, and a config file that is vulnerable to the regression would report linting errors in V1. You can quickly verify that your configs are interpreted without regression by comparing the output of `benthos -c ./yourconfig.yaml --print-yaml` with V1 and V2. If they are the same then you are not affected. ### [](#field-default-value-changes)Field default value changes In version 2 the field `unsubscribe_on_close` of the `nats_stream` input is now `false` by default. ## [](#service)Service The recommended way to create plugins for Redpanda Connect is outlined in [this repository](https://github.com/benthosdev/benthos-plugin-example). Therefore the following experimental plugin related flags have been removed from the service: `swap-envs`, `plugins-dir`, `list-input-plugins`, `list-output-plugins`, `list-processor-plugins`, `list-condition-plugins`. The flag `swap-envs` has also been removed for clarity, as it had no impact on JSON reference resolution. If this flag is being used please open an issue and it can be reimplemented to be fully compliant. ## [](#go-api)Go API ### [](#condition-package-moved)Condition package moved The package `github.com/Jeffail/benthos/lib/processor/condition` has been changed to `github.com/Jeffail/benthos/lib/condition`. Migrating should be a simple case of applying a find/replace on your codebase: ```sh find . -name "*.go" | \ xargs sed -i 's/benthos\/lib\/processor\/condition/benthos\/lib\/condition/g' ``` ### [](#interface-changes)Interface changes The following interface changes have occurred to core Redpanda Connect components: - `types.Cache` now has `types.Closable` embedded. - `types.RateLimit` now has `types.Closable` embedded. - `types.Manager` has new method `GetPlugin`. - `log.Modular` has new method `WithFields`. --- # Page 380: Migrating to Version 3 **URL**: https://docs.redpanda.com/redpanda-connect/guides/migration/v3.md --- # Migrating to Version 3 --- title: Migrating to Version 3 latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: migration/v3 page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: migration/v3.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/migration/v3.adoc description: Learn how to migrate to Redpanda Connect v3. page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- Redpanda Connect version 3 comes with some breaking [service](#service) and [configuration](#configuration) changes as well as a few breaking [API changes](#go-api). ## [](#service)Service ### [](#memory-map-file-buffer-removed)Memory map file buffer removed The long deprecated `mmap_file` buffer has been removed. If you were still relying on this buffer implementation then please [raise an issue](https://github.com/redpanda-data/connect/issues). ### [](#old-metrics-paths-removed)Old metrics paths removed The following undocumented metrics paths have been removed: - `input.parts.count` - `input.read.success` - `input.read.error` - `input.send.success` - `input.send.error` - `input.ack.success` - `input.ack.error` - `output.running` - `output.parts.count` - `output.send.success` - `output.parts.send.success` - `output.send.error` All of these paths have [remaining equivalents](../../../components/metrics/about/#paths). ### [](#metric-path-changes)Metric path changes The `http_client` output client metrics have been renamed from `output.**.output.http_client**` **to `output.`**`.client`. ## [](#configuration)Configuration ### [](#metrics-prefix)Metrics prefix The configuration field `prefix` within `metrics` has been moved from the root of the config object to individual types. E.g. when using `statsd` the field `metrics.prefix` should be replaced with `metrics.statsd.prefix`. ### [](#json-paths)JSON paths Many components within Redpanda Connect use an unspecified "JSON dot path" syntax for querying and setting fields within JSON documents. The format of these paths has been formalized to make them clearer and more generally useful, but this potentially breaks your paths when they query against hierarchies that contain arrays. The formal specification for v3 can be found [in this document](../../../configuration/field_paths/). The following components are affected: - `awk` processor (all of the `json_*` functions) - `json` processor (`path` field) - `process_field` processor (`path` field) - `process_map` processor (`premap`, `premap_optional`, `postmap` and `postmap_optional` fields) - `check_field` condition (`path` field) - `json_field` function interpolation - `s3` input (`sqs_body_path`, `sqs_bucket_path` and `sqs_envelope_path` fields) - `dynamodb` output (`json_map_columns` field values) #### [](#migration-guide)Migration guide In order to replicate the exact same behavior as currently exists your paths should be updated to include the character `*` wherever an array exists. For example, the default value of `sqs_body_path` for the `s3` input has been updated from `Records.s3.object.key` to `Records.*.s3.object.key`. ### [](#process-dag-stage-names)Process DAG stage names The `process_dag` processor now only permits workflow stage names matching the following regular expression: `[a-zA-Z0-9_-]+`. The reasoning for this restriction is to potentially expand the features of `process_dag` in the future with custom root fields (e.g. `$on_error`). ## [](#go-api)Go API ### [](#modules)Modules Redpanda Connect now fully adheres to [Go Modules](https://github.com/golang/go/wiki/Modules), import paths must therefore now contain the major version (v3) like so: ```go import "github.com/Jeffail/benthos/v3/lib/processor" ``` It should be pretty quick to update your imports, either using a tool or just: ```sh grep "Jeffail/benthos" . -Rl | grep -e "\.go$" | xargs -I{} sed -i 's/Jeffail\/benthos/Jeffail\/benthos\/v3/g' {} ``` ### [](#other)Other - Constructors for buffer components now require a `types.Manager`, giving them parity with other components: `buffer.New(conf Config, mgr types.Manager, log log.Modular, stats metrics.Type) (Type, error)` --- # Page 381: Migrating to Version 4 **URL**: https://docs.redpanda.com/redpanda-connect/guides/migration/v4.md --- # Migrating to Version 4 --- title: Migrating to Version 4 latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: migration/v4 page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: migration/v4.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/migration/v4.adoc description: Learn how to migrate to Redpanda Connect v4, with insights into new functionalities and migration tips. page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- Redpanda Connect has been at major version 3 [for more than two years](/blog/2021/01/04/v4-roadmap), during which time it has gained a huge amount of functionality without introducing any breaking changes. However, the number of components, APIs and features that have been deprecated in favour of better solutions has grown steadily and the time has finally come to purge them. There are also some areas of functionality that have been improved with breaking changes. This document outlines the changes made to Redpanda Connect since V3 and tips for how to migrate to V4 in places where those changes are significant. ## [](#deprecated-components-removed)Deprecated components removed All components, features and configuration fields that were marked as deprecated in the latest release of V3 have been removed in V4. In order to detect deprecated components or fields within your existing configuration files you can run the linter from a later release of V3 Redpanda Connect with the `--deprecated` flag: ```sh benthos lint --deprecated ./configs/*.yaml ``` This should report all remaining deprecated components. All deprecated components have favoured alternative solutions in V3, so it should be possible to slowly eliminate deprecated aspects of your config using V3 before upgrading. ### [](#unit-test-directories)Unit test directories The `benthos test` subcommand no longer walks paths when they are directories. Instead use explicit triple-dot syntax (`./dir/...`) or wildcard patterns. ## [](#new-go-module-name)New Go module name For users of the Go plugin APIs the import path of this module needs to be updated to `github.com/benthosdev/benthos/v4`, like so: ```go import "github.com/benthosdev/benthos/v4/public/service" ``` ## [](#pulsar-components-disabled-for-now)Pulsar components disabled (for now) There have been multiple issues with the Go Pulsar client libraries. Since some are still outstanding and causing problems with unrelated components the decision has been made to remove the `pulsar` input and output from standard builds. However, it is still possible to build custom versions of Redpanda Connect with them included by importing the package `./public/components/pulsar`: ```go package main import ( "context" "github.com/Jeffail/benthos/v3/public/service" // Import all plugins defined within the repo. _ "github.com/benthosdev/benthos/v4/public/components/all" _ "github.com/benthosdev/benthos/v4/public/components/pulsar" ) func main() { service.RunCLI(context.Background()) } ``` ## [](#pipeline-threads-behavior-change)Pipeline threads behavior change In V3 the `pipeline.threads` field defaults to 1. If this field is explicitly set to `0` it will automatically match the number of CPUs on the host machine. In V4 this will change so that the default value of `pipeline.threads` is `-1`, where this value indicates we should match the number of host CPUs. An explicit value of `0` is still considered valid and functionally equivalent to `-1`. [Related issue](https://github.com/redpanda-data/connect/issues/399). ## [](#old-style-interpolation-functions-removed)Old Style Interpolation Functions Removed The original style of interpolation functions, where you specify a function name followed by a colon and then any arguments (`${!json:foo,1}`) has been deprecated (and undocumented) for a while now. What we’ve had instead is a subset of Bloblang allowing you to use functions directly (`${! json("foo").from(1) }`), but with the old style still supported for backwards compatibility. However, supporting the old style means our parsing capabilities are weakened and so it is now removed in order to allow more powerful interpolations in the future. ## [](#bloblang-changes)Bloblang changes The functions `meta`, `root_meta`, `error` and `env` now return `null` when the target value does not exist. This is in order to improve consistency across different functions and query types. In cases where a default empty string is preferred you can add `.or("")` onto the function. In cases where you want to throw an error when the value does not exist you can add `.not_null()` onto the function. [Related issue](https://github.com/redpanda-data/connect/issues/571). ### [](#root-referencing)Root referencing It is now possible to reference the `root` of the document being created within a mapping query, i.e. `root.hash = root.string().hash("xxhash64")`. ## [](#env-var-docker-configuration)Env var Docker configuration Docker builds will no longer come with a default config that contains generated environment variables. This system doesn’t scale at all for complex configuration files and was becoming a challenge to maintain (and also huge). Instead, the new `-s` flag has been the preferred way to configure Redpanda Connect through arguments and will need to be used exclusively in V4. It’s worth noting that this does not prevent you from defining your own env var based configuration and adding that to your docker image. It’s entirely possible to copy the config from V3 and have that work, it just won’t be present by default any more. In order to migrate to the `-s` flag use the path of the fields you’re setting instead of the generated environment variables, so: ```sh docker run --rm -p 4195:4195 jeffail/benthos \ -e "INPUT_TYPE=http_server" \ -e "OUTPUT_TYPE=kafka" \ -e "OUTPUT_KAFKA_ADDRESSES=kafka-server:9092" \ -e "OUTPUT_KAFKA_TOPIC=benthos_topic" ``` Becomes: ```sh docker run --rm -p 4195:4195 jeffail/benthos \ -s "input.type=http_server" \ -s "output.type=kafka" \ -s "output.kafka.addresses=kafka-server:9092" \ -s "output.kafka.topic=benthos_topic" ``` ## [](#old-plugin-apis-removed)Old plugin APIs removed Any packages from within the `lib` directory have been removed. Please use only the APIs within the `public` directory, the API docs count be found on [pkg.go.dev](https://pkg.go.dev/github.com/benthosdev/benthos/v4/public), and examples can be found in the [`benthos-plugin-example` repository](https://github.com/benthosdev/benthos-plugin-example). These new APIs can be found in V3 so if you have many components you can migrate them incrementally by sticking with V3 until completion. Many of the old packages within `lib` can also still be found within `internal`, if you’re in a pickle you can find some of those APIs and copy/paste them into your own repository. ## [](#caches)Caches All caches that support retries have had their retry/backoff configuration fields modified in order to be more consistent. The new common format is: ```yml retries: initial_interval: 1s max_interval: 5s max_elapsed_time: 30s ``` In cases where it might be desirable to disable retries altogether (the `ristretto` cache) there is also an `enabled` field. ### [](#ttl-changes)TTL changes Caches that support TTLs have had their `ttl` fields renamed to `default_ttl` in order to make it clearer that their purpose is to provide a fallback. All of these values are now duration string types, i.e. a cache with an integer seconds based field with a previous value of `60` should now be defined as `60s`. ## [](#field-default-changes)Field default changes Lots of fields have had default values removed in cases where they were deemed unlikely to be useful and likely to cause frustration. This specifically applies to any `url`, `urls`, `address` or `addresses` fields that may have once had a default value containing a common example for the particular service. In most cases this should cause minimal disruption as the field is non-optional and therefore not specifying it explicitly will result in config errors. [Related issue](https://github.com/redpanda-data/connect/issues/392). However, there are the following exceptions that are worth noting: ### [](#the-http-processor-and-http_client-output-no-longer-create-multipart-requests-by-default)The `http` processor and `http_client` output no longer create multipart requests by default The `http` processor and `http_client` output now execute message batch requests as individual requests by default. This behavior can be disabled by explicitly setting `batch_as_multipart` to `true`. ### [](#output-lines-codec-no-longer-adds-extra-batch-newlines)Output `lines` codec no longer adds extra batch newlines All outputs that traditionally wrote empty newlines at the end of batches with >1 message when using the `lines` codec (`socket`, `stdout`, `file`, `sftp`) no longer do this by default. This was originally kept for backwards compatibility but was often seen as an unexpected and annoying behavior. It is still possible to add these end-of-batch newlines in a more consistent way by either adding an empty message to the end of batches, or by adding a newline to the last message of the batch. ### [](#the-switch-output-retry_until_success)The `switch` output `retry_until_success` By default the `switch` output continues retrying switch case outputs until success. This default was sensible at the time as we didn’t have a concept of intentionally nacking messages, and therefore a nacked message was likely a recoverable problem and retrying internally means that messages matching multiple cases wouldn’t produce duplicates. However, since then Redpanda Connect has evolved and a very common pattern with the `switch` output is to reject messages that failed during processing using the `reject` output. But because of the default value of `retry_until_success` many users end up in a confusing situation where using a `reject` output results in the pipeline blocking indefinitely until they discover this field. Therefore the default value of `retry_until_success` will now be `false`, which means users that aren’t using a `reject` flow in one of their switch cases, and have a configuration where messages could match multiple cases, should explicitly set this field to `true` in order to avoid potential duplicates during downstream outages. ### [](#aws-region-fields)AWS `region` fields Any configuration sections containing AWS fields no longer have a default `region` of `eu-west-1`. Instead, the field will be empty by default, where unless explicitly set the environment variable `AWS_REGION` will be used. This will cause problems for users where they expect the region `eu-west-1` to be targeted when neither the field nor the environment variable `AWS_REGION` are set. [Related issue](https://github.com/redpanda-data/connect/issues/696). ## [](#clickhouse-driver-changes)Clickhouse driver changes The `clickhouse` SQL driver Data Source Name format parameters have been changed due to a client library update (details can be found at [https://github.com/ClickHouse/clickhouse-go](https://github.com/ClickHouse/clickhouse-go)). A compatibility layer has been added that makes a best attempt to translate the old DSN format to the new one, but some parameters may not carry over exactly. This update also means placeholders in `sql_raw` queries should be in dollar syntax. ## [](#serverless-default-output)Serverless default output The default output of the serverless distribution of Redpanda Connect is now the following config: ```yml output: switch: retry_until_success: false cases: - check: errored() output: reject: "processing failed due to: ${! error() }" - output: sync_response: {} ``` This change was made in order to return processing errors directly to the invoker by default. ## [](#metrics-changes)Metrics changes The metrics produced by a Redpanda Connect stream have been greatly simplified and now make better use of labels/tags in order to provide component-specific insights. The configuration and behavior of metrics types has also been made more consistent, with metric names being the same throughout and `mapping` now being a general top-level field. For a full overview of the new system check out the [metrics about page](../../../components/metrics/about/). [Related issue](https://github.com/redpanda-data/connect/issues/1066). ### [](#field-prefix-is-gone)Field `prefix` is gone Some metrics components such as `prometheus` had a `prefix` field for setting a prefix to all metric names. These fields are now gone, if you want to reintroduce these prefixes you can use the general purpose `mapping` field. For example, if we previously had a config: ```yml metrics: prometheus: prefix: ${METRICS_PREFIX:benthos} ``` We need to delete that prefix and add a mapping that renames metric names: ```yaml metrics: mapping: 'root = env("METRICS_PREFIX").or("benthos") + "_" + this' prometheus: {} ``` ### [](#the-http_server-type-renamed-to-json_api)The `http_server` type renamed to `json_api` The name given to the generic JSON API metrics type was `http_server`, which was confusing as it isn’t the only metrics output type that presents as an HTTP server endpoint. This type was also only originally intended for local debugging, which the `prometheus` type is also good for. In order to distinguish this metrics type by its unique feature, which is that it exposes metrics as a JSON object, it has been renamed to `json_api`. ### [](#the-stdout-type-renamed-to-logger)The `stdout` type renamed to `logger` The `stdout` metrics type now emits metrics using the Redpanda Connect logger, and therefore also matches the logger format. As such, it has been renamed to `logger` in order to reflect that. ### [](#no-more-dots)No more dots In V3 metrics names contained dots in order to represent pseudo-paths of the source component. In V4 all metric names produced by Redpanda Connect have been changed to contain only alpha-numeric characters and underscores. It is recommended that any custom metric names produced by your `metric` processors and custom plugins should match this new format for consistency. Since dots were invalid characters in Prometheus metric names, in V3 the `prometheus` metrics type made some automatic modifications to all names before registering them. This rewrite first replaced all `-` and _characters to a double underscore (`_`_), and then replaced all `.` characters with `_`. This was an ugly work around and has been removed in V4, but means in previous cases where custom metrics containing dots were automatically converted you will instead see error logs reporting that the names were invalid and therefore ignored. If you wish to retain the old rewrite behavior you can reproduce it with the new `mapping` field: ```yml metrics: mapping: 'root = this.replace("_", "__").replace("-", "__").replace(".", "_")' prometheus: {} ``` However, it’s recommended to change your metric names instead. ## [](#tracing-changes)Tracing changes Distributed tracing within Redpanda Connect is now done via the Open Telemetry client library. Unfortunately, this client library does not support the full breadth of options as we had before. As such, the `jaeger` tracing type now only supports the `const` sampling type, and the field `service_name` has been removed. This will likely mean tracing output will appear different in this release, and if you were relying on code that extracts and interacts with spans from messages in your custom plugins then it will need to be converted to use the official Open Telemetry APIs. [Related issue](https://github.com/redpanda-data/connect/issues/872). ## [](#logging-changes)Logging changes The `logger` config section has been simplified, the new default set to `logfmt`, and the `classic` format removed. The default value of `add_timestamp` has also been changed to `false`. [Related issue](https://github.com/redpanda-data/connect/issues/589). ## [](#automatic-max-in-flight)Automatic max In flight Outputs that compose other outputs (`broker`, `switch`, etc) no longer require their own `max_in_flight` settings as they will automatically saturate their composed outputs. This includes outputs that compose resources. ## [](#processor-batch-behavior-changes)Processor batch behavior changes Some processors that once executed only once per batch have been updated to execute upon each message individually by default. This change has been made because it was felt the individual message case was considerably more common (and intuitive) and that it is possible to satisfy the batch-wide behavior in other ways that are opt-in, such as by placing the processors within a `branch` and having your `request_map` explicit for a single `batch_index` (i.e. `request_map: root = if batch_index() != 0 { deleted() }`). [Related issue](https://github.com/redpanda-data/connect/issues/408). ### [](#processor-parts-field-removed)Processor `parts` field removed Many processors previously had a `parts` field, which allowed you to explicitly list the indexes of a batch to apply the processor to. This field had confusing naming and was rarely used (or even known about). Since that same behavior can be reproduced by placing the processor within a `branch` (or `switch`) all `parts` fields have been removed. ### [](#dedupe)`dedupe` The `dedupe` processor has been reworked so that it now acts upon individual messages by default. It’s now mandatory to specify a `key`, and the `parts` and `hash` fields have been removed. Instead, specify full-content hashing with interpolation functions in the `key` field, e.g. `${! content().hash("xxhash64") }`. In order to deduplicate an entire batch it is likely easier to use a `cache` processor with the `add` operator: ```yml pipeline: processors: # Try and add one message to a cache that identifies the whole batch - branch: request_map: | root = if batch_index() == 0 { this.id } else { deleted() } processors: - cache: operator: add key: ${! content() } value: t # Delete all messages if we failed - mapping: | root = if errored().from(0) { deleted() } ``` ### [](#log)`log` The `log` processor now executes for every message of batches by default. ### [](#sleep)`sleep` The `sleep` processor now executes for every message of batches by default. ## [](#broker-ditto-macro-gone)Broker ditto macro gone The hidden macro `ditto` for broker configs is now removed. Use the `copies` field instead. For some edge cases where `copies` does not satisfy your requirements you may be better served using [configuration templates](../../../configuration/templating/). If all else fails then please [reach out](https://redpanda.com/slack) and we can look into other solutions. --- # Page 382: Monitoring **URL**: https://docs.redpanda.com/redpanda-connect/guides/monitoring.md --- # Monitoring --- title: Monitoring latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: monitoring page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: monitoring.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/monitoring.adoc description: Learn how to monitor Redpanda Connect, including setting up metrics and alerts to ensure system health and performance. page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- ## [](#health-checks)Health checks Redpanda Connect serves two HTTP endpoints for health checks: - `/ping` can be used as a liveness probe as it always returns a 200. - `/ready` can be used as a readiness probe as it serves a 200 only when both the input and output are connected, otherwise a 503 is returned. ## [](#metrics)Metrics Redpanda Connect [exposes lots of metrics](../../components/metrics/about/#metric_names) either to Statsd, Prometheus, Cloudwatch or for debugging purposes an HTTP endpoint that returns a JSON formatted object. The target destination of Redpanda Connect metrics is configurable from the [metrics section](../../components/metrics/about/), where it’s also possible to rename and restrict the metrics that are emitted with mappings. ## [](#tracing)Tracing Redpanda Connect also [emits opentracing events](../../components/tracers/about/) to a tracer of your choice, which can be used to visualise the processors within a pipeline. --- # Page 383: Performance Tuning **URL**: https://docs.redpanda.com/redpanda-connect/guides/performance_tuning.md --- # Performance Tuning --- title: Performance Tuning latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: performance_tuning page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: performance_tuning.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/performance_tuning.adoc description: Discover best practices for tuning Redpanda Connect performance, optimizing configurations, and achieving maximum throughput. page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- ## [](#maximising-io-throughput)Maximising IO throughput This section outlines a few common throughput issues and ways in which they can be solved within Redpanda Connect. It is assumed here that your Redpanda Connect instance is performing only minor processing steps, and therefore has minimal reliance on your CPU resource. If this is not the case the following still applies to an extent, but you should also refer to [the next section regarding CPU utilization](#maximising-cpu-utilization). Firstly, before venturing into Redpanda Connect configurations, you should take an in-depth look at your sources and sinks. Redpanda Connect is generally much simpler architecturally than the inputs and outputs it supports. Spend some time understanding how to squeeze the most out of these services and it will make it easier (or unnecessary) to tune your Redpanda Connect configuration. ### [](#redpanda-connect-reads-too-slowly)Redpanda Connect reads too slowly If Redpanda Connect isn’t reading fast enough from your source it might not necessarily be due to a slow consumer. If the sink is slow this can cause back pressure that throttles the amount Redpanda Connect can read. Try consuming a test feed with the output replaced with `drop`. If you notice that the input consumption suddenly speeds up then the issue is likely with the output, in which case [try the next section](#benthos-writes-too-slowly). If the `drop` output pipe didn’t help then take a quick look at the basic configuration fields for the input source type. Sometimes there are fields for setting a number of background prefetches or similar concepts that can increase your throughput. For example, increasing the value of `prefetch_count` for an AMQP consumer can greatly increase the rate at which it is consumed. Next, if your source supports multiple parallel consumers then you can try doing that within Redpanda Connect by using a [broker](../../components/inputs/broker/). For example, if you started with: ```yaml input: http_client: url: http://localhost:4195/get verb: GET ``` You could change to: ```yaml input: broker: copies: 4 inputs: - http_client: url: http://localhost:4195/get verb: GET ``` Which would create the exact same consumer as before with four connections in total. Try increasing the number of copies to see how that affects the throughput. If your multiple consumers would require different configurations then set copies to `1` and write each consumer as a separate object in the `inputs` array. Read the [broker documentation](../../components/inputs/broker/) for more tips on simplifying broker configs. If your source doesn’t support multiple parallel consumers then unfortunately your options are more limited. A logical next step might be to look at your network/disk configuration to see if that’s a potential cause of contention. ### [](#redpanda-connect-writes-too-slowly)Redpanda Connect Writes Too Slowly If you have an output sink that regularly places back pressure on your source there are a few solutions depending on the details of the issue. Firstly, you should check the config parameters of your output sink. There are often fields specifically for controlling the level of acknowledgement to expect before moving onto the next message, if these levels of guarantee are overkill you can disable them for greater throughput. For example, setting the `ack_replicas` field to `false` in the Kafka sink can have a high impact on throughput. If the config parameters for an output sink aren’t enough then you can try the following: #### [](#increase-in-flight-messages)Increase in flight messages Most outputs have a field `max_in_flight` that allows you to specify how many messages can be in flight at the same time. Increasing this value can improve throughput significantly. #### [](#send-messages-in-batches)Send messages in batches Most outputs will send data quicker when messages are batched, this is often done automatically in the background. However, for a few outputs your batches need to be configured. Read the [batching documentation](../../configuration/batching/) for more guidance on how to tune message batches within Redpanda Connect. #### [](#level-out-input-spikes-with-a-buffer)Level out input spikes with a buffer There are many reasons why an input source might have spikes or inconsistent throughput rates. It is possible that your output is capable of keeping up with the long term average flow of data, but fails to keep up when an intermittent spike occurs. In situations like these it is sometimes a better use of your hardware and resources to level out the flow of data rather than try and match the peak throughput. This would depend on the frequency and duration of the spikes as well as your latency requirements, and is therefore a matter of judgement. Leveling out the flow of data can be done within Redpanda Connect using a [buffer](../../components/buffers/about/). Buffers allow an input source to store a bounded amount of data temporarily, which a consumer can work through at its own pace. Buffers always have a fixed capacity, which when full will proceed to block the input just like a busy output would. Therefore, it’s still important to have an output that can keep up with the flow of data, the difference that a buffer makes is that the output only needs to keep up with the _average_ flow of data versus the instantaneous flow of data. For example, if your input usually produces 10 msgs/s, but occasionally spikes to 100 msgs/s, and your output can handle up to 50 msgs/s, it might be possible to configure a buffer large enough to store spikes in their entirety. As long as the average flow of messages from the input remains below 50 msgs/s then your service should be able to continue indefinitely without ever blocking the input source. ## [](#maximising-cpu-utilization)Maximising CPU utilization Some [processors](../../components/processors/about/) within Redpanda Connect are relatively heavy on your CPU, and can potentially become the bottleneck of a service. In these circumstances it is worth configuring Redpanda Connect so that your processors are running on each available core of your machine without contention. An array of processors in any section of a Redpanda Connect config becomes a single logical pipeline of steps running on a single logical thread. The easiest way to create parallel processor threads is to configure them inside the [pipeline](../../configuration/processing_pipelines/) configuration block, where we can explicitly set any number of parallel processor threads independent of how many inputs or outputs we want to use. Please refer [to the documentation regarding pipelines](../../configuration/processing_pipelines/) for some examples. --- # Page 384: Serverless **URL**: https://docs.redpanda.com/redpanda-connect/guides/serverless/about.md --- # Serverless --- title: Serverless latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: serverless/about page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: serverless/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/serverless/about.adoc description: Learn about serverless architecture in Redpanda Connect, including benefits, use cases, and implementation details. page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- Currently the only supported serverless deployment of Redpanda Connect is for [AWS Lambda](../lambda/), if you are interested in other platforms please [raise an issue](https://github.com/redpanda-data/connect/issues). ## [](#platforms)Platforms - [AWS Lambda](../lambda/) --- # Page 385: Lambda **URL**: https://docs.redpanda.com/redpanda-connect/guides/serverless/lambda.md --- # Lambda --- title: Lambda latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: serverless/lambda page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: serverless/lambda.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/serverless/lambda.adoc description: Deploying as an AWS Lambda function page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- You can use either of the following distributions to deploy Redpanda Connect as an AWS Lambda function: - The `redpanda-connect-lambda` distribution is specifically tailored for deployment as an AWS Lambda function on the `go1.x` runtime, which runs Amazon Linux on the `x86_64` architecture. - The `redpanda-connect-lambda-al2` distribution supports the `provided.al2` runtime, which runs Amazon Linux 2 on either the `x86_64` or `arm64` architecture. ## [](#configuration)Configuration The AWS Lambda version of Redpanda Connect uses the same configuration format as a regular instance. Choose from the following two options: 1. Inline via the `BENTHOS_CONFIG` environment variable (YAML format). 2. Via the filesystem using a layer, extension, or container image. By default, the `redpanda-connect-lambda` distribution looks for a valid configuration file in the locations listed below. Alternatively, you can set the configuration file path explicity by passing a `BENTHOS_CONFIG_PATH` environment variable. - `./benthos.yaml` - `./config.yaml` - `/benthos.yaml` - `/etc/benthos/config.yaml` - `/etc/benthos.yaml` Also, the `http`, `input` and `buffer` sections are ignored as the service wide HTTP server is not used, and messages are inserted via function invocations. If the `output` section is omitted in your config then the result of the processing pipeline is returned back to the caller, otherwise the resulting data is sent to the output destination. ## [](#run-with-an-output)Run with an output The flow of a Redpanda Connect Lambda function with an output configured looks like this: ```text redpanda-connect-lambda +------------------------------+ | | -------> Processors ----> Output -----> Somewhere invoke | | | <-------------------------------------------/ | | | | +------------------------------+ ``` Where the call will block until the output target has confirmed receipt of the resulting payload. When the message is successfully propagated a JSON payload is returned of the form `{"message":"request successful"}`, otherwise an error is returned containing the reason for the failure. ## [](#run-without-an-output)Run without an output The flow when an output is not configured looks like this: ```text redpanda-connect-lambda +--------------------+ | | -------> Processors --\ | invoke | | | <---------------------/ | | | | | +--------------------+ ``` Where the function returns the result of processing directly back to the caller. The format of the result differs depending on the number of batches and messages of a batch that resulted from the invocation: - Single message of a single batch: `{}` (JSON object) - Multiple messages of a single batch: `[{},{}]` (Array of JSON objects) - Multiple batches: `[[{},{}],[{}]]` (Array of arrays of JSON objects, batches of size one are a single object array in this case) ### [](#process-errors)Process errors By default, the Redpanda Connect Lambda handler fails if messages encounter an uncaught error during execution. ## [](#run-a-combination)Run a combination It’s possible to configure pipelines that send messages to third party destinations and also return a result back to the caller. This is done by configuring an output block and including an output of the type `sync_response`. For example, if we wished for our Lambda function to send a payload to Kafka and also return the same payload back to the caller we could use a [broker](../../../components/outputs/broker/): ```yml output: broker: pattern: fan_out outputs: - kafka: addresses: - todo:9092 client_id: benthos_serverless topic: example_topic - sync_response: {} ``` ## [](#upload-to-aws)Upload to AWS ### [](#go1-x-on-x86_64)go1.x on x86\_64 Grab an archive labelled `redpanda-connect-lambda` from the [releases page](https://github.com/redpanda-data/connect/releases) page and then create your function: ```sh LAMBDA_ENV=`cat yourconfig.yaml | jq -csR {Variables:{BENTHOS_CONFIG:.}}` aws lambda create-function \ --runtime go1.x \ --handler redpanda-connect-lambda \ --role benthos-example-role \ --zip-file fileb://redpanda-connect-lambda_4.87.0_linux_amd64.zip \ --environment "$LAMBDA_ENV" \ --function-name benthos-example ``` ### [](#provided-al2-on-arm64)provided.al2 on arm64 Grab an archive labelled `redpanda-connect-lambda-al2` for `arm64` from the [releases page](https://github.com/redpanda-data/connect/releases) page and then create your function (AWS CLI v2 only): ```sh LAMBDA_ENV=`cat yourconfig.yaml | jq -csR {Variables:{BENTHOS_CONFIG:.}}` aws lambda create-function \ --runtime provided.al2 \ --architectures arm64 \ --handler not.used.for.provided.al2.runtime \ --role benthos-example-role \ --zip-file fileb://redpanda-connect-lambda-al2_4.87.0_linux_arm64.zip \ --environment "$LAMBDA_ENV" \ --function-name benthos-example ``` Note that you can also run `redpanda-connect-lambda-al2` on `x86_64`, just use the `amd64` zip instead. ## [](#invoke)Invoke ```sh aws lambda invoke \ --function-name benthos-example \ --payload '{"your":"document"}' \ out.txt && cat out.txt && rm out.txt ``` ## [](#build)Build You can build and archive the function yourself with: ```sh go build github.com/redpanda-data/connect/v4/cmd/serverless/connect-lambda zip connect-lambda.zip connect-lambda ``` --- # Page 386: Streams Mode **URL**: https://docs.redpanda.com/redpanda-connect/guides/streams_mode/about.md --- # Streams Mode --- title: Streams Mode latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: streams_mode/about page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: streams_mode/about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/streams_mode/about.adoc description: Get an overview of streams mode in Redpanda Connect, detailing its features, use cases, and setup instructions. page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- A Redpanda Connect stream consists of four components; an input, an optional buffer, processor pipelines and an output. Under normal use a Redpanda Connect instance is a single stream, and these components are configured within the service config file. Alternatively, Redpanda Connect can be run in `streams` mode, where a single running Redpanda Connect instance is able to run multiple entirely isolated streams. Adding streams in this mode can be done in two ways: 1. [Static configuration files](../using_config_files/) allows you to maintain a directory of static stream configuration files that will be traversed by Redpanda Connect. 2. An [HTTP REST API](../using_rest_api/) allows you to dynamically create, read the status of, update, and delete streams at runtime. These two methods can be used in combination, i.e. it’s possible to update and delete streams that were created with static files. When running Redpanda Connect in streams mode it is still necessary to provide a general service wide configuration with the `-o`/`--observability` flag that specifies observability configuration such as the `metrics`, `logger` and `tracing` sections, as well the `http` section for configuring how the HTTP server should behave. You can import resources either in the general configuration, or using the `-r`/`--resources` flag, the same as when running Redpanda Connect in regular mode. ```bash rpk connect streams -r "./prod/*.yaml" -o ./config.yaml ``` ## [](#http-endpoints)HTTP endpoints A Redpanda Connect config can contain components such as an `http_server` input that register endpoints to the service-wide HTTP server. When these components are created from within a named stream in streams mode the endpoint will be prefixed with the streams identifier by default. For example, a stream with the identifier `foo` and the config: ```yaml input: http_server: path: /meow pipeline: processors: - mapping: 'root = "meow " + content()' output: sync_response: {} ``` Will register an endpoint `/meow`, which will be prefixed with the name `foo` to become `/foo/meow`. This behavior is intended to make a clearer distinction between endpoints registered by different streams, and prevent collisions of those endpoints. However, you can disable this behavior by setting the flag `--prefix-stream-endpoints` to `false` (`rpk connect streams --prefix-stream-endpoints=false ./streams/*.yaml`). ## [](#resources)Resources When running Redpanda Connect in streams mode [resource components](../../../configuration/resources/) are shared across all streams. The streams mode HTTP API also provides an endpoint for modifying and adding resource configurations dynamically. ## [](#metrics)Metrics Metrics from all streams are aggregated and exposed via the method specified in [the config](../../../components/metrics/about/) of the Redpanda Connect instance running in `streams` mode, with their metrics enriched with the tag `stream` containing the stream name. For example, a Redpanda Connect instance running in streams mode running a stream named `foo` would have metrics from `foo` registered with the label `stream` with the value of `foo`. This can cause problems if your streams are short lived and uniquely named as the number of metrics registered will continue to climb indefinitely. In order to avoid this you can use the `mapping` field to filter metric names. ```yaml # Only register metrics for the stream `foo`. Others will be ignored. metrics: mapping: if meta("stream") != "foo" { deleted() } prometheus: {} ``` --- # Page 387: Streams API **URL**: https://docs.redpanda.com/redpanda-connect/guides/streams_mode/streams_api.md --- # Streams API --- title: Streams API latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: streams_mode/streams_api page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: streams_mode/streams_api.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/guides/pages/streams_mode/streams_api.adoc description: Get an overview of streams mode in Redpanda Connect, detailing its features, use cases, and setup instructions. page-git-created-date: "2024-05-24" page-git-modified-date: "2024-09-05" --- When Redpanda Connect is run in `streams` mode it will open up an HTTP REST API for creating and managing independent streams of data instead of creating a single stream. Each stream has its own input, buffer, pipeline and output sections which contains an isolated stream of data with its own lifetime. A stream config cannot include [resources](../../../configuration/resources/), and instead these should be created and modified using the `/resources/{type}/{id}` endpoint. A walkthrough on using this API [can be found here](../using_rest_api/). ## [](#api)API ### [](#get-ready)GET `/ready` Returns a 200 OK response if all active streams are connected to their respective inputs and outputs at the time of the request. Otherwise, a 503 response is returned along with a message naming the faulty stream. If zero streams are active this endpoint still returns a 200 OK response. ### [](#get-streams)GET `/streams` Returns a map of existing streams by their unique identifiers to an object showing their status and uptime. #### [](#response-200)Response 200 ```json { "": { "active": "", "uptime": "", "uptime_str": "" } } ``` ### [](#post-streams)POST `/streams` Sets the entire collection of streams to the body of the request. Streams that exist but aren’t within the request body are _removed_, streams that exist already and are in the request body are updated, other streams within the request body are created. ```json { "": "" } ``` #### [](#response-200-2)Response 200 The streams were updated successfully. #### [](#response-400)Response 400 A configuration was invalid, or has linting errors. If linting errors were detected then a JSON response is provided of the form: ```json { "linting_errors": [ "", "uptime": "", "uptime_str": "", "config": "" } ``` ### [](#put-streamsid)PUT `/streams/{id}` Update an existing stream identified by `id` by posting a body containing the new stream configuration in either JSON or YAML format. The configuration should be a standard Redpanda Connect configuration containing the sections `input`, `buffer`, `pipeline` and `output`. The previous stream will be shut down before and a new stream will take its place. #### [](#response-200-5)Response 200 The stream was updated successfully. #### [](#response-400-3)Response 400 The configuration was invalid, or has linting errors. If linting errors were detected then a JSON response is provided of the form: ```json { "linting_errors": [ " ./streams/foo.yaml < ./streams/bar.yaml < Input (AMQP) -> Processors -> Output (AMQP) <------- Acknowledgement --------- ``` However, Redpanda Connect supports bidirectional protocols like HTTP and WebSocket, which allow responses to be returned directly from the pipeline. For example, HTTP is a request/response protocol, and inputs like `http_server` (Self-Managed) or `gateway` (Redpanda Cloud) support returning response payloads to the requester. ```text --------- Request Body --------> Input (HTTP) -> Processors -> Output (Sync Response) <--- Response Body (and ack) --- ``` ## [](#routing-processed-messages-back)Routing processed messages back To return a processed response, use the [`sync_response`](../../components/outputs/sync_response/) output. Use the `http_server` input in Self-Managed deployments: ```yaml input: http_server: path: /weather pipeline: processors: - mapping: | root = { city: json("location"), forecast: "Clear skies with light winds", temperature_c: 22 } output: sync_response: {} ``` Sending this request: ```json { "location": "Berlin" } ``` Returns: ```json { "city": "Berlin", "forecast": "Clear skies with light winds", "temperature_c": 22 } ``` ## [](#combine-with-other-outputs)Combine with other outputs You can route processed messages to storage and return a response using a [`broker`](../../components/outputs/broker/) output. ```yaml input: http_server: path: /weather output: broker: pattern: fan_out outputs: - kafka: addresses: [localhost:9092] topic: weather.requests - sync_response: processors: - mapping: | root = { status: "received", received_at: now() } ``` ## [](#returning-partially-processed-messages)Returning partially processed messages You can return a response before the message is fully processed by using the [`sync_response` processor](../../components/processors/sync_response/). This allows continued processing after the response is set. ```yaml pipeline: processors: - mapping: root = "Received weather report for %s".format(json("location")) - sync_response: {} - mapping: root.reported_at = now() ``` This returns `"Received weather report for Berlin"` to the client, but continues modifying the message before storing or forwarding it. > 📝 **NOTE** > > Due to delivery guarantees, the response is not sent until all downstream processing and acknowledgements are complete. ## [](#routing-output-responses-back)Routing output responses back Some outputs, such as [`http_client`](../../components/outputs/http_client/), support returning a downstream response back to the input. ```yaml input: http_server: path: /proxy-weather output: http_client: url: https://api.example.com/forecast verb: POST propagate_response: true ``` This forwards the request to an external weather API and returns its response to the original requester. You can combine this with other outputs: ```yaml input: http_server: path: /proxy-weather output: broker: pattern: fan_out outputs: - kafka: addresses: [localhost:9092] topic: weather.proxy - http_client: url: https://api.example.com/forecast verb: POST propagate_response: true ``` --- # Page 391: Redpanda Connect Documentation **URL**: https://docs.redpanda.com/redpanda-connect/home.md --- # Redpanda Connect Documentation --- title: Redpanda Connect Documentation latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: index page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: index.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/home/pages/index.adoc description: Home page for the Redpanda Connect docs. page-git-created-date: "2024-12-13" page-git-modified-date: "2024-12-13" --- ## Overview Redpanda Connect is a data streaming service for building scalable, high-performance data pipelines that drive real-time analytics and actionable business insights. Integrate data across systems with hundreds of prebuilt connectors, change data capture (CDC) capabilities, and YAML-configurable pipelines. [Learn more](../get-started/about/) ## Deploy[](#home-primary-title) [ ### Cloud Start using Redpanda Connect in Redpanda Cloud. Get started ](../../redpanda-cloud/develop/connect/connect-quickstart/) --- # Page 392: Install or Upgrade Redpanda Connect **URL**: https://docs.redpanda.com/redpanda-connect/install.md --- # Install or Upgrade Redpanda Connect --- title: Install or Upgrade Redpanda Connect latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: index page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: index.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/install/pages/index.adoc description: Discover all ways to install or upgrade Redpanda Connect. page-git-created-date: "2025-09-25" page-git-modified-date: "2025-09-25" --- This page helps you choose how to install or upgrade Redpanda Connect. Select the method that best fits your needs: | Method | When to use | Limitations | | --- | --- | --- | | Redpanda CLI (recommended) | The simplest way to install, upgrade, and manage Redpanda Connect. Works on all supported platforms. Automatically upgrades with a single command. | Doesn’t include connectors that require cgo (for example, TigerBeetle). | | Helm chart (Kubernetes) | Deploy and manage Redpanda Connect on Kubernetes clusters using Helm. Supports upgrades and rollbacks. | Requires a running Kubernetes cluster and Helm. Not for bare metal or VM installs. | | Prebuilt binary (advanced users) | Use when you need a cgo-enabled binary with extra connectors, such as TigerBeetle. Prebuilt archives are published for Linux AMD64. | Not available for Windows, macOS, or Docker images. You must manage upgrades manually. | | Build from source (power users / contributors) | Gives full control. Lets you build with CGO_ENABLED=1 to include connectors such as TigerBeetle. | Requires Go toolchain and build setup. | --- # Page 393: Build or Upgrade from Source **URL**: https://docs.redpanda.com/redpanda-connect/install/build-from-source.md --- # Build or Upgrade from Source --- title: Build or Upgrade from Source latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: build-from-source page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: build-from-source.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/install/pages/build-from-source.adoc description: Build or upgrade Redpanda Connect from source. page-git-created-date: "2025-09-25" page-git-modified-date: "2025-09-25" --- You can build Redpanda Connect in two ways: - **Standard build**: Suitable for most users. Does not require cgo. Does not include the [TigerBeetle CDC input connector](../../components/inputs/tigerbeetle_cdc/). - **cgo-enabled build**: Required only if you need the TigerBeetle CDC input connector. ## [](#install-from-source)Install from source 1. Ensure you have Go installed on your system. You can download it from [the official Go website](https://golang.org/dl/). 2. Clone the Redpanda Connect repository: ```bash git clone https://github.com/redpanda-data/connect.git cd connect ``` 3. Build the binary (choose one): | Build type | Build command | | --- | --- | | Standard (no cgo, most users) | go build -o redpanda-connect ./cmd/redpanda-connect | | cgo-enabled (TigerBeetle CDC support) | CGO_ENABLED=1 go build -o redpanda-connect ./cmd/redpanda-connect | 4. Move the binary to a directory in your `PATH` (for example, `/usr/local/bin`): ```bash sudo mv redpanda-connect /usr/local/bin/ ``` 5. Verify the installation: ```bash redpanda-connect --version ``` You should see a version. ## [](#upgrade-a-source-build)Upgrade a source build 1. Navigate to your existing clone of the repository: ```bash cd connect ``` 2. Fetch the latest tags and check out the version you want: ```bash git fetch --all git checkout v4.87.0 ``` 3. Rebuild the binary (choose one): | Build type | Build command | | --- | --- | | Standard (no cgo, most users) | go build -o redpanda-connect ./cmd/redpanda-connect | | cgo-enabled (TigerBeetle CDC support) | CGO_ENABLED=1 go build -o redpanda-connect ./cmd/redpanda-connect | 4. Replace the previous binary and verify: ```bash sudo mv redpanda-connect /usr/local/bin/ redpanda-connect --version ``` > 📝 **NOTE** > > After upgrading (or downgrading), you must restart a pipeline to use features in the Redpanda Connect version you have deployed. ## [](#uninstall)Uninstall To uninstall Redpanda Connect, remove the installed binary: ```sh sudo rm /usr/local/bin/redpanda-connect ``` This removes the Redpanda Connect binary from your system. ## [](#next-steps)Next steps Explore the available connectors in the [catalog](../../components/about/). --- # Page 394: Install or Upgrade with the Helm Chart **URL**: https://docs.redpanda.com/redpanda-connect/install/helm-chart.md --- # Install or Upgrade with the Helm Chart --- title: Install or Upgrade with the Helm Chart latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: helm-chart page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: helm-chart.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/install/pages/helm-chart.adoc description: Deploy or upgrade Redpanda Connect on Kubernetes using Helm. page-git-created-date: "2025-09-25" page-git-modified-date: "2025-09-25" --- The official Redpanda Connect Helm chart lets you deploy and manage Redpanda Connect on Kubernetes. ## [](#install-with-helm)Install with Helm To deploy Redpanda Connect on Kubernetes, use the official Helm chart: ```bash helm repo add redpanda https://charts.redpanda.com helm repo update helm install redpanda-connect redpanda/connect --namespace --create-namespace ``` Replace `` with your desired Kubernetes namespace. > 💡 **TIP** > > See [Get Started with the Redpanda Connect Helm Chart](../../get-started/quickstarts/helm-chart/) for a full walkthrough and configuration options. ## [](#upgrade-with-helm)Upgrade with Helm To upgrade Redpanda Connect to a newer Helm chart version: 1. Ensure your Helm repository is updated to fetch the latest version of the Redpanda Helm chart: ```bash helm repo update ``` 2. Upgrade your existing deployment using the [`helm upgrade`](https://helm.sh/docs/helm/helm_upgrade/) command: ```bash helm upgrade redpanda-connect redpanda/connect --namespace --values ``` Replace `` with the path to your custom configuration file. ### [](#roll-back-an-upgrade)Roll back an upgrade If an upgrade introduces issues, you can roll back to a previous release. Helm retains previous deployments, allowing you to roll back with the [`helm rollback`](https://helm.sh/docs/helm/helm_rollback/) command: ```bash helm rollback redpanda-connect --namespace ``` To find the revision number, use the [`helm history`](https://helm.sh/docs/helm/helm_history/) command: ```bash helm history redpanda-connect --namespace ``` This command lists previous deployments and their revision numbers. ## [](#uninstall)Uninstall To uninstall Redpanda Connect from your Kubernetes cluster: ```bash helm uninstall redpanda-connect --namespace ``` Replace `` with the namespace where Redpanda Connect was installed. ## [](#next-steps)Next steps Explore the available connectors in the [catalog](../../components/about/). --- # Page 395: Install or Upgrade from a Prebuilt Binary **URL**: https://docs.redpanda.com/redpanda-connect/install/prebuilt-binary.md --- # Install or Upgrade from a Prebuilt Binary --- title: Install or Upgrade from a Prebuilt Binary latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: prebuilt-binary page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: prebuilt-binary.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/install/pages/prebuilt-binary.adoc description: Download, install, or upgrade Redpanda Connect from a prebuilt binary. page-git-created-date: "2025-09-25" page-git-modified-date: "2025-09-25" --- You can choose between two types of prebuilt binaries: - **Standard binary**: Suitable for most users. Does not require cgo. Does not include the [TigerBeetle CDC input connector](../../components/inputs/tigerbeetle_cdc/). - **cgo-enabled binary**: Required only if you need the TigerBeetle CDC input connector. Only available for Linux AMD64. ## [](#install-a-prebuilt-binary)Install a prebuilt binary 1. Ensure you have `wget` and `tar` installed. 2. Download the desired Redpanda Connect binary for your platform. You can find a complete list of releases on the [GitHub releases page](https://github.com/redpanda-data/connect/releases). | Binary type | Example download command | | --- | --- | | Standard (no cgo, most users) | wget https://github.com/redpanda-data/redpanda-connect/releases/download/v4.87.0/redpanda-connect_4.87.0_linux_amd64.tar.gz | | cgo-enabled (TigerBeetle CDC support) | wget https://github.com/redpanda-data/redpanda-connect/releases/download/v4.87.0/redpanda-connect-cgo_4.87.0_linux_amd64.tar.gz | 3. Extract the binary from the archive (replace the filename as needed): ```bash tar -xzf .tar.gz ``` 4. Move the binary to a directory in your `PATH` (for example, `/usr/local/bin`): ```bash sudo mv redpanda-connect /usr/local/bin/ ``` 5. Verify the installation: ```bash redpanda-connect --version ``` You should see a version number. ## [](#upgrade-a-prebuilt-binary)Upgrade a prebuilt binary To upgrade, repeat the installation steps with the new version. Stop any running Redpanda Connect processes, download and extract the new binary, replace the old binary, and verify the upgrade. > 📝 **NOTE** > > After upgrading (or downgrading), you must restart a pipeline to use features in the Redpanda Connect version you have deployed. ## [](#uninstall)Uninstall To uninstall Redpanda Connect, remove the installed binary: ```sh sudo rm /usr/local/bin/redpanda-connect ``` This removes the Redpanda Connect binary from your system. ## [](#next-steps)Next steps Explore the available connectors in the [catalog](../../components/about/). --- # Page 396: Install or Upgrade with rpk **URL**: https://docs.redpanda.com/redpanda-connect/install/rpk.md --- # Install or Upgrade with rpk --- title: Install or Upgrade with rpk latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: rpk page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: rpk.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/install/pages/rpk.adoc description: Install or upgrade Redpanda Connect using the Redpanda CLI. page-git-created-date: "2025-09-25" page-git-modified-date: "2025-09-25" --- The Redpanda CLI (`rpk`) is the recommended way to install, upgrade, and manage Redpanda Connect on bare metal. It provides a simple, automated workflow for most users. ## [](#install)Install To install Redpanda Connect using the Redpanda CLI, choose your platform: ### [](#linux)Linux To install, or update to, the latest version of `rpk` for Linux, run: #### amd64 ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-amd64.zip -d ~/.local/bin/ ``` #### arm64 ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-arm64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-arm64.zip -d ~/.local/bin/ ``` > 💡 **TIP** > > You can use `rpk` on Windows only with [WSL](https://learn.microsoft.com/windows/wsl/install). However, commands that require Redpanda to be installed on your machine are not supported, such as [`rpk container`](../../../current/reference/rpk/rpk-container/rpk-container/) commands, [`rpk iotune`](../../../current/reference/rpk/rpk-iotune/), and [`rpk redpanda`](../../../current/reference/rpk/rpk-redpanda/rpk-redpanda/) commands. To install, or update to, a version other than the latest, run: #### amd64 ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/download/v/rpk-linux-amd64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-amd64.zip -d ~/.local/bin/ ``` #### arm64 ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/download/v/rpk-linux-arm64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-linux-arm64.zip -d ~/.local/bin/ ``` #### [](#fips-compliance)FIPS compliance > 📝 **NOTE** > > This feature requires an [enterprise license](../../get-started/licensing/). You can either [upgrade to an Enterprise Edition license](https://www.redpanda.com/upgrade), or [generate a trial license key](http://redpanda.com/try-enterprise) that’s valid for 30 days. To install `rpk` to run the latest version of Redpanda Connect in FIPS-compliant mode, you must install the `redpanda-rpk-fips` and `redpanda-connect-fips` packages. Both packages are built using the [Microsoft GoLang compiler](https://github.com/microsoft/go) and the [Microsoft’s Go Crypto OpenSSL package](https://github.com/microsoft/go-crypto-openssl), which uses the FIPS-approved version of OpenSSL. > ❗ **IMPORTANT** > > The packages for FIPS-compliant `rpk` (`redpanda-rpk-fips`) and Redpanda `rpk` (`redpanda-rpk`) are mutually exclusive, and so cannot be installed in the same environment. ##### RHEL 1. To make sure your distribution is up to date, run: ```bash sudo dnf upgrade ``` 2. Add `redpanda` to your `dnf` list of repositories. ```bash curl -1sLf 'https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.rpm.sh' | \ sudo -E bash ``` 3. Install Redpanda packages for FIPS compliance. ```bash sudo dnf install -y redpanda-rpk-fips redpanda-connect-fips ``` 4. Verify your installation. ```bash rpk connect --version ``` To keep up-to-date with Redpanda Connect releases, run the following command: sudo dnf update ##### Debian/Ubuntu 1. To make sure your distribution is up to date, run: ```bash sudo apt upgrade ``` 2. Add `redpanda` to your `apt` list of repositories. ```bash curl -1sLf 'https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.deb.sh' | sudo -E bash ``` 3. Install Redpanda packages for FIPS compliance. ```bash sudo apt install -y redpanda-rpk-fips redpanda-connect-fips ``` 4. Verify your installation. ```bash rpk connect --version ``` To keep up-to-date with the Redpanda Connect releases, run the following command: sudo apt update ### [](#macos)MacOS #### Homebrew 1. If you don’t have Homebrew installed, [install it](https://brew.sh/). 2. To install or update `rpk`, run: ```bash brew install redpanda-data/tap/redpanda ``` #### Manual Download To install or update `rpk` through a manual download, choose the option for your system architecture. For example, if you have an M1 or newer chip, select **Apple Silicon**. ##### Intel macOS To install, or update to, the latest version of `rpk` for Intel macOS, run: ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-darwin-amd64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-darwin-amd64.zip -d ~/.local/bin/ ``` To install, or update to, a version other than the latest, run: ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/download/v/rpk-darwin-amd64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-darwin-amd64.zip -d ~/.local/bin/ ``` ##### Apple Silicon To install, or update to, the latest version of `rpk` for Apple Silicon, run: ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-darwin-arm64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-darwin-arm64.zip -d ~/.local/bin/ ``` To install, or update to, a version other than the latest, run: ```bash curl -LO https://github.com/redpanda-data/redpanda/releases/download/v/rpk-darwin-arm64.zip && mkdir -p ~/.local/bin && export PATH="~/.local/bin:$PATH" && unzip rpk-darwin-arm64.zip -d ~/.local/bin/ ``` ### [](#docker)Docker To use `rpk` in Docker, you can use the Redpanda Docker image, which includes both Redpanda and `rpk` as part of the installation. ```bash docker pull docker.redpanda.com/redpandadata/connect ``` ## [](#upgrade)Upgrade Redpanda Connect uses the `rpk connect` plugin to manage upgrades, which is automatically installed when you start to run `rpk connect` commands from the `rpk` CLI. You do not need to upgrade the entire Redpanda CLI. Check the version of Redpanda Connect already installed: ```bash rpk connect --version ``` ### [](#upgrade-to-the-latest-version)Upgrade to the latest version The simplest way to upgrade is to use the `rpk connect upgrade` command: ```bash rpk connect upgrade ``` Example output: ```bash Redpanda Connect successfully upgraded from 4.38.0 to the latest version (4.87.0). ``` For a full list of `rpk connect upgrade` options, run: ```bash rpk connect upgrade --help ``` ### [](#upgrade-to-a-specific-version)Upgrade to a specific version Force an upgrade (or downgrade) to a specific version of Redpanda Connect using the following command: ```bash rpk connect install --connect-version --force ``` Replace the `` placeholder with the full version number, for example: `4.38.0`. You may need to wait a few seconds before the installation completes. Example output: ```bash Redpanda Connect 4.38.0 successfully installed. ``` ## [](#uninstall)Uninstall If you no longer want to use Redpanda Connect, you can uninstall the `rpk connect` plugin from the CLI: ```bash rpk connect uninstall ``` Example output: ```bash PATH MESSAGE /Users/username/.local/bin/.rpk.managed-connect OK ``` > 📝 **NOTE** > > After upgrading (or downgrading), you must restart a pipeline to use features in the Redpanda Connect version you have deployed. ## [](#next-steps)Next steps Explore the available connectors in the [catalog](../../components/about/). --- # Page 397: Plugins **URL**: https://docs.redpanda.com/redpanda-connect/plugins/about.md --- # Plugins --- title: Plugins latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: about page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: about.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/plugins/pages/about.adoc description: Create and load dynamic plugins at runtime with gRPC. page-git-created-date: "2025-06-11" page-git-modified-date: "2025-06-11" --- Redpanda Connect’s RPC plugin framework is a powerful Apache 2.0 licensed feature that takes data streaming flexibility to the next level. By enabling you to create and load plugins at runtime using any programming language that supports gRPC, this framework opens up a world of new integration possibilities beyond Go. Whether you need to leverage specialized libraries, integrate with AI/ML tools, or build custom components in your preferred language, Redpanda Connect plugins provide the extensibility and interoperability to make it happen. ## [](#plugin-types)Plugin types Redpanda Connect supports two types of plugins: Compiled plugins Built in Go and compiled directly into the Redpanda Connect binary. These offer maximum performance for critical workloads. Dynamic plugins External executables that communicate with Redpanda Connect via gRPC. These provide language flexibility and can be loaded at runtime without rebuilding the binary. This documentation focuses on dynamic plugins. ## [](#when-to-use-dynamic-plugins)When to use dynamic plugins Dynamic plugins are ideal when you need to extend Redpanda Connect’s capabilities without the constraints of Go. Use cases include: - Use existing libraries not available in Go - Write plugins in languages other than Go (such as Python) - Add plugins without rebuilding the entire Redpanda Connect binary - Deploy plugins independently of your main Redpanda Connect deployment - Integrate with AI/ML frameworks, data science libraries, or specialized tools ## [](#architecture)Architecture Dynamic plugins run as separate processes and communicate with the main Redpanda Connect process through gRPC over Unix sockets. This architecture provides the following benefits: - Process isolation: Plugin crashes don’t affect the main engine. - Language agnostic: Support for any language with gRPC libraries. - Modular design: Each plugin maps to a single subprocess. - Battle-tested communication. Uses proven gRPC technology. The system includes three compiled plugin wrappers (one for each component type: `BatchInput`, `BatchProcessor`, and `BatchOutput`) that handle the communication with external plugin executables. ## [](#supported-languages)Supported languages Redpanda Connect currently provides the following SDKs: - **Go**: Familiar environment for existing Redpanda Connect developers with type-safe interfaces - **Python**: Access to rich ecosystem including PyTorch, TensorFlow, Hugging Face Transformers, LangChain, Pandas, and NumPy ## [](#performance-considerations)Performance considerations Dynamic plugins introduce some serialization and IPC overhead compared to compiled plugins. To minimize performance impact: - The system uses batch components exclusively to amortize cross-process communication costs. - Each plugin runs in a separate process with predictable resource usage. - For maximum performance in critical workloads, use compiled plugins instead. ## [](#prerequisites)Prerequisites Before creating dynamic plugins, ensure you have: - Redpanda Connect v4.56.0 or later - Python environment (for Python plugins) - Required language SDKs installed ### [](#install-redpanda-connect)Install Redpanda Connect Verify your version, and upgrade if necessary: ```shell rpk --version rpk connect upgrade ``` ## [](#create-your-first-plugin)Create your first plugin This example demonstrates creating a simple FizzBuzz processor plugin in Python. ### [](#step-1-set-up-the-environment)Step 1: Set up the environment ```shell # Create a new directory for your plugin mkdir fizzbuzz_plugin cd fizzbuzz_plugin # Initialize with link:https://docs.astral.sh/uv/[uv] (or use pip/conda) uv init . uv add redpanda_connect ``` ### [](#step-2-download-the-example)Step 2: Download the example ```shell # Download the example processor curl -o fizzbuzz_processor.py \ https://raw.githubusercontent.com/redpanda-data/connect/main/public/plugin/python/examples/fizzbuzz_processor.py ``` ### [](#step-3-create-the-plugin-configuration)Step 3: Create the plugin configuration Create `plugin.yaml`: ```yaml name: fizzbuzz summary: Transforms numbers according to FizzBuzz rules (3=Fizz, 5=Buzz, 15=FizzBuzz) command: ["uv", "run", "fizzbuzz_processor.py"] type: processor fields: [] ``` ### [](#step-4-create-the-pipeline-configuration)Step 4: Create the pipeline configuration Create `connect.yaml`: ```yaml input: generate: interval: 1s mapping: | root = counter() % 20 + 1 # Generate numbers 1-20 pipeline: processors: - fizzbuzz: {} output: stdout: codec: lines ``` ### [](#step-5-run-the-plugin)Step 5: Run the plugin Execute the pipeline with your dynamic plugin: ```shell rpk connect run --rpc-plugins=plugin.yaml connect.yaml ``` Expected output: ```text 1 2 Fizz 4 Buzz Fizz 7 8 Fizz Buzz 11 Fizz 13 14 FizzBuzz 16 17 Fizz 19 Buzz ``` ## [](#python-plugin-development)Python plugin development ### [](#basic-processor-structure)Basic processor structure Python processors use the `@redpanda_connect.processor` decorator: ```python import asyncio import logging import redpanda_connect @redpanda_connect.processor def transform_message(msg: redpanda_connect.Message) -> redpanda_connect.Message: # Your transformation logic here msg.payload = msg.payload.upper() return msg if __name__ == "__main__": logging.basicConfig(level=logging.INFO) asyncio.run(redpanda_connect.processor_main(transform_message)) ``` ### [](#plugin-configuration-structure)Plugin configuration structure Each plugin requires a configuration file with the following structure: ```yaml name: summary: command: ["", ""] type: fields: [] # Configuration fields (if any) ``` ### [](#message-handling)Message handling The Python SDK provides access to message properties: - `msg.payload` - Message content - `msg.metadata` - Message metadata - Standard message transformation methods ## [](#next-steps)Next steps - Explore the [example plugins](https://github.com/redpanda-data/connect/tree/main/public/plugin/python/examples) in the Redpanda Connect repository - Join the [Redpanda Community Slack](https://redpanda.com/slack) to discuss plugin development - Review the [Redpanda Connect source code](https://github.com/redpanda-data/connect) for advanced use cases ## [](#related-topics)Related topics - [Components overview](../../components/about/) - [Configuration overview](../../configuration/about/) - [Getting started with Redpanda Connect](../../get-started/quickstarts/rpk/) ## Suggested labs - [Flatten JSON Messages](/redpanda-labs/data-transforms/flatten-go/) - [Convert JSON Messages into Avro](/redpanda-labs/data-transforms/issdemo-go/) - [Convert Timestamps using Rust](/redpanda-labs/data-transforms/ts-converter-rust/) - [Filter Messages into a New Topic using a Regex](/redpanda-labs/data-transforms/regex-go/) - [Redact Information in JSON Messages](/redpanda-labs/data-transforms/redaction-go/) [Search all labs](/redpanda-labs) --- # Page 398: Reference **URL**: https://docs.redpanda.com/redpanda-connect/reference.md --- # Reference --- title: Reference latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: index page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: index.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/reference/pages/index.adoc description: The following topics provide reference material for Redpanda Connect. page-git-created-date: "2024-11-14" page-git-modified-date: "2024-11-14" --- - [Redpanda Connect Chart Specification](k-connect-helm-spec/) --- # Page 399: Redpanda Connect Chart Specification **URL**: https://docs.redpanda.com/redpanda-connect/reference/k-connect-helm-spec.md --- # Redpanda Connect Chart Specification --- title: Redpanda Connect Chart Specification latest-connect-version: 4.87.0 categories: input: - name: Services description: Inputs that consume from storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/inputs/amqp_1/ status: community - name: Amazon DynamoDB CDC url: /redpanda-connect/components/inputs/aws_dynamodb_cdc/ status: certified - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/inputs/beanstalkd/ status: community - name: Apache Cassandra url: /redpanda-connect/components/inputs/cassandra/ status: community - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: git url: /redpanda-connect/components/inputs/git/ status: certified - name: hdfs url: /redpanda-connect/components/inputs/hdfs/ status: community - name: microsoft_sql_server_cdc url: /redpanda-connect/components/inputs/microsoft_sql_server_cdc/ status: certified - name: MongoDB url: /redpanda-connect/components/inputs/mongodb/ status: certified - name: MongoDB CDC url: /redpanda-connect/components/inputs/mongodb_cdc/ status: certified - name: mqtt url: /redpanda-connect/components/inputs/mqtt/ status: certified - name: mysql_cdc url: /redpanda-connect/components/inputs/mysql_cdc/ status: certified - name: NATS url: /redpanda-connect/components/inputs/nats/ status: certified - name: NATS KV url: /redpanda-connect/components/inputs/nats_kv/ status: certified - name: NATS JetStream url: /redpanda-connect/components/inputs/nats_jetstream/ status: certified - name: NATS Stream url: /redpanda-connect/components/inputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/inputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/inputs/ockam_kafka/ status: community - name: Oracle CDC url: /redpanda-connect/components/inputs/oracledb_cdc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: postgres_cdc url: /redpanda-connect/components/inputs/postgres_cdc/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/inputs/pulsar/ status: community - name: Redis List url: /redpanda-connect/components/inputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/inputs/redis_pubsub/ status: certified - name: Redis url: /redpanda-connect/components/inputs/redis_scan/ status: certified - name: Redis Streams url: /redpanda-connect/components/inputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/inputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/inputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/inputs/redpanda_migrator/ status: certified - name: spicedb_watch url: /redpanda-connect/components/inputs/spicedb_watch/ status: community - name: splunk url: /redpanda-connect/components/inputs/splunk/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/inputs/sql_select/ status: certified - name: tigerbeetle_cdc url: /redpanda-connect/components/inputs/tigerbeetle_cdc/ status: certified - name: timeplus url: /redpanda-connect/components/inputs/timeplus/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Network description: Inputs that consume directly from low level network protocols. items: - name: HTTP url: /redpanda-connect/components/inputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/inputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/inputs/nanomsg/ status: community - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/inputs/otlp_http/ status: certified - name: sftp url: /redpanda-connect/components/inputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/inputs/socket/ status: certified - name: socket_server url: /redpanda-connect/components/inputs/socket_server/ status: certified - name: websocket url: /redpanda-connect/components/inputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/inputs/zmq4/ status: community - name: AWS description: Inputs that consume from Amazon Web Services products. items: - name: AWS CloudWatch Logs url: /redpanda-connect/components/inputs/aws_cloudwatch_logs/ status: community - name: AWS Kinesis url: /redpanda-connect/components/inputs/aws_kinesis/ status: certified - name: AWS S3 url: /redpanda-connect/components/inputs/aws_s3/ status: certified - name: AWS SQS url: /redpanda-connect/components/inputs/aws_sqs/ status: certified - name: GCP description: Inputs that consume from Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/inputs/gcp_bigquery_select/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/inputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/inputs/gcp_pubsub/ status: certified - name: Google Cloud url: /redpanda-connect/components/inputs/gcp_spanner_cdc/ status: certified - name: Azure description: Inputs that consume from Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/inputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/inputs/azure_cosmosdb/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/inputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/inputs/azure_table_storage/ status: certified - name: Social description: Inputs that consume from social applications and services. items: - name: discord url: /redpanda-connect/components/inputs/discord/ status: community - name: twitter_search url: /redpanda-connect/components/inputs/twitter_search/ status: community - name: Local description: Inputs that consume from the local machine/filesystem. items: - name: CSV url: /redpanda-connect/components/inputs/csv/ status: certified - name: file url: /redpanda-connect/components/inputs/file/ status: certified - name: parquet url: /redpanda-connect/components/inputs/parquet/ status: community - name: stdin url: /redpanda-connect/components/inputs/stdin/ status: certified - name: Utility description: Inputs that provide utility by generating data or combining/wrapping other inputs. items: - name: batched url: /redpanda-connect/components/inputs/batched/ status: certified - name: broker url: /redpanda-connect/components/inputs/broker/ status: certified - name: dynamic url: /redpanda-connect/components/inputs/dynamic/ status: community - name: generate url: /redpanda-connect/components/inputs/generate/ status: certified - name: inproc url: /redpanda-connect/components/inputs/inproc/ status: certified - name: read_until url: /redpanda-connect/components/inputs/read_until/ status: certified - name: resource url: /redpanda-connect/components/inputs/resource/ status: certified - name: sequence url: /redpanda-connect/components/inputs/sequence/ status: certified - name: subprocess url: /redpanda-connect/components/inputs/subprocess/ status: community buffer: - name: Windowing description: Buffers that provide message windowing capabilities. items: - name: system_window url: /redpanda-connect/components/buffers/system_window/ status: certified - name: Utility description: Buffers that are intended for niche but general use. items: - name: memory url: /redpanda-connect/components/buffers/memory/ status: certified - name: sqlite url: /redpanda-connect/components/buffers/sqlite/ status: community processor: - name: Mapping description: Processors that specialize in restructuring messages. items: - name: awk url: /redpanda-connect/components/processors/awk/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: javascript url: /redpanda-connect/components/processors/javascript/ status: certified - name: jmespath url: /redpanda-connect/components/processors/jmespath/ status: certified - name: jq url: /redpanda-connect/components/processors/jq/ status: certified - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: JSON Schema url: /redpanda-connect/components/processors/json_schema/ status: certified - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: Integration description: Processors that interact with external services. items: - name: Amazon url: /redpanda-connect/components/processors/aws_dynamodb_partiql/ status: certified - name: AWS Lambda url: /redpanda-connect/components/processors/aws_lambda/ status: certified - name: cache url: /redpanda-connect/components/processors/cache/ status: certified - name: command url: /redpanda-connect/components/processors/command/ status: certified - name: couchbase url: /redpanda-connect/components/processors/couchbase/ status: community - name: GCP BigQuery url: /redpanda-connect/components/processors/gcp_bigquery_select/ status: certified - name: http url: /redpanda-connect/components/processors/http/ status: certified - name: redis url: /redpanda-connect/components/processors/redis/ status: certified - name: Redis Script url: /redpanda-connect/components/processors/redis_script/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_raw/ status: certified - name: SQL url: /redpanda-connect/components/processors/sql_select/ status: certified - name: subprocess url: /redpanda-connect/components/processors/subprocess/ status: community - name: Parsing description: Processors that specialize in translating messages from one format to another. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: avro url: /redpanda-connect/components/processors/avro/ status: community - name: bloblang url: /redpanda-connect/components/processors/bloblang/ status: certified - name: compress url: /redpanda-connect/components/processors/compress/ status: certified - name: decompress url: /redpanda-connect/components/processors/decompress/ status: certified - name: grok url: /redpanda-connect/components/processors/grok/ status: community - name: mapping url: /redpanda-connect/components/processors/mapping/ status: certified - name: msgpack url: /redpanda-connect/components/processors/msgpack/ status: community - name: mutation url: /redpanda-connect/components/processors/mutation/ status: certified - name: parquet_decode url: /redpanda-connect/components/processors/parquet_decode/ status: certified - name: parquet_encode url: /redpanda-connect/components/processors/parquet_encode/ status: certified - name: parse_log url: /redpanda-connect/components/processors/parse_log/ status: community - name: protobuf url: /redpanda-connect/components/processors/protobuf/ status: certified - name: schema_registry_decode url: /redpanda-connect/components/processors/schema_registry_decode/ status: certified - name: schema_registry_encode url: /redpanda-connect/components/processors/schema_registry_encode/ status: certified - name: string_split url: /redpanda-connect/components/processors/string_split/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: xml url: /redpanda-connect/components/processors/xml/ status: community - name: Composition description: Higher level processors that compose other processors and modify their behavior. items: - name: branch url: /redpanda-connect/components/processors/branch/ status: certified - name: catch url: /redpanda-connect/components/processors/catch/ status: certified - name: for_each url: /redpanda-connect/components/processors/for_each/ status: certified - name: group_by url: /redpanda-connect/components/processors/group_by/ status: certified - name: group_by_value url: /redpanda-connect/components/processors/group_by_value/ status: certified - name: insert_part url: /redpanda-connect/components/processors/insert_part/ status: certified - name: parallel url: /redpanda-connect/components/processors/parallel/ status: certified - name: processors url: /redpanda-connect/components/processors/processors/ status: certified - name: retry url: /redpanda-connect/components/processors/retry/ status: certified - name: switch url: /redpanda-connect/components/processors/switch/ status: certified - name: try url: /redpanda-connect/components/processors/try/ status: certified - name: while url: /redpanda-connect/components/processors/while/ status: certified - name: workflow url: /redpanda-connect/components/processors/workflow/ status: certified - name: Utility description: Processors that provide general utility or do not fit in another category. items: - name: ZIP url: /redpanda-connect/components/processors/archive/ status: certified - name: benchmark url: /redpanda-connect/components/processors/benchmark/ status: certified - name: bounds_check url: /redpanda-connect/components/processors/bounds_check/ status: certified - name: cached url: /redpanda-connect/components/processors/cached/ status: certified - name: crash url: /redpanda-connect/components/processors/crash/ status: certified - name: dedupe url: /redpanda-connect/components/processors/dedupe/ status: certified - name: log url: /redpanda-connect/components/processors/log/ status: certified - name: metric url: /redpanda-connect/components/processors/metric/ status: certified - name: rate_limit url: /redpanda-connect/components/processors/rate_limit/ status: certified - name: redpanda_data_transform url: /redpanda-connect/components/processors/redpanda_data_transform/ status: certified - name: resource url: /redpanda-connect/components/processors/resource/ status: certified - name: select_parts url: /redpanda-connect/components/processors/select_parts/ status: certified - name: sleep url: /redpanda-connect/components/processors/sleep/ status: certified - name: split url: /redpanda-connect/components/processors/split/ status: certified - name: sync_response url: /redpanda-connect/components/processors/sync_response/ status: certified - name: ZIP url: /redpanda-connect/components/processors/unarchive/ status: certified - name: wasm url: /redpanda-connect/components/processors/wasm/ status: community output: - name: Services description: Outputs that write to storage or message streaming services. items: - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_0_9/ status: certified - name: RabbitMQ url: /redpanda-connect/components/outputs/amqp_1/ status: community - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: beanstalkd url: /redpanda-connect/components/outputs/beanstalkd/ status: community - name: cache url: /redpanda-connect/components/outputs/cache/ status: certified - name: cypher url: /redpanda-connect/components/outputs/cypher/ status: community - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: elasticsearch_v8 url: /redpanda-connect/components/outputs/elasticsearch_v8/ status: certified - name: elasticsearch_v9 url: /redpanda-connect/components/outputs/elasticsearch_v9/ status: community - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: hdfs url: /redpanda-connect/components/outputs/hdfs/ status: community - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: MongoDB url: /redpanda-connect/components/outputs/mongodb/ status: certified - name: mqtt url: /redpanda-connect/components/outputs/mqtt/ status: certified - name: NATS url: /redpanda-connect/components/outputs/nats/ status: certified - name: NATS JetStream url: /redpanda-connect/components/outputs/nats_jetstream/ status: certified - name: NATS KV url: /redpanda-connect/components/outputs/nats_kv/ status: certified - name: NATS Stream url: /redpanda-connect/components/outputs/nats_stream/ status: community - name: nsq url: /redpanda-connect/components/outputs/nsq/ status: community - name: ockam_kafka url: /redpanda-connect/components/outputs/ockam_kafka/ status: community - name: opensearch url: /redpanda-connect/components/outputs/opensearch/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_grpc/ status: certified - name: OpenTelemetry url: /redpanda-connect/components/outputs/otlp_http/ status: certified - name: Apache Pulsar url: /redpanda-connect/components/outputs/pulsar/ status: community - name: pusher url: /redpanda-connect/components/outputs/pusher/ status: community - name: questdb url: /redpanda-connect/components/outputs/questdb/ status: certified - name: Redis Hash url: /redpanda-connect/components/outputs/redis_hash/ status: certified - name: Redis List url: /redpanda-connect/components/outputs/redis_list/ status: certified - name: Redis PubSub url: /redpanda-connect/components/outputs/redis_pubsub/ status: certified - name: Redis Streams url: /redpanda-connect/components/outputs/redis_streams/ status: certified - name: redpanda url: /redpanda-connect/components/outputs/redpanda/ status: certified - name: redpanda_common url: /redpanda-connect/components/outputs/redpanda_common/ status: certified - name: redpanda_migrator url: /redpanda-connect/components/outputs/redpanda_migrator/ status: certified - name: Salesforce url: /redpanda-connect/components/outputs/salesforce_sink/ status: certified - name: Snowflake url: /redpanda-connect/components/outputs/snowflake_put/ status: certified - name: Snowflake Streaming url: /redpanda-connect/components/outputs/snowflake_streaming/ status: certified - name: Splunk url: /redpanda-connect/components/outputs/splunk_hec/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_insert/ status: certified - name: SQL url: /redpanda-connect/components/outputs/sql_raw/ status: certified - name: timeplus url: /redpanda-connect/components/outputs/timeplus/ status: community - name: Network description: Outputs that write directly to low level network protocols. items: - name: HTTP url: /redpanda-connect/components/outputs/http_client/ status: certified - name: HTTP url: /redpanda-connect/components/outputs/http_server/ status: certified - name: nanomsg url: /redpanda-connect/components/outputs/nanomsg/ status: community - name: sftp url: /redpanda-connect/components/outputs/sftp/ status: certified - name: socket url: /redpanda-connect/components/outputs/socket/ status: certified - name: websocket url: /redpanda-connect/components/outputs/websocket/ status: certified - name: zmq4 url: /redpanda-connect/components/outputs/zmq4/ status: community - name: AWS description: Outputs that write to Amazon Web Services products. items: - name: AWS DynamoDB url: /redpanda-connect/components/outputs/aws_dynamodb/ status: community - name: AWS Kinesis url: /redpanda-connect/components/outputs/aws_kinesis/ status: certified - name: AWS Kinesis Firehose url: /redpanda-connect/components/outputs/aws_kinesis_firehose/ status: certified - name: AWS S3 url: /redpanda-connect/components/outputs/aws_s3/ status: certified - name: AWS SNS url: /redpanda-connect/components/outputs/aws_sns/ status: community - name: AWS SQS url: /redpanda-connect/components/outputs/aws_sqs/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: GCP description: Outputs that write to Google Cloud Platform services. items: - name: GCP BigQuery url: /redpanda-connect/components/outputs/gcp_bigquery/ status: certified - name: GCP Cloud Storage url: /redpanda-connect/components/outputs/gcp_cloud_storage/ status: certified - name: GCP PubSub url: /redpanda-connect/components/outputs/gcp_pubsub/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Azure description: Outputs that write to Microsoft Azure services. items: - name: Azure Blob Storage url: /redpanda-connect/components/outputs/azure_blob_storage/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_cosmosdb/ status: certified - name: Microsoft Azure url: /redpanda-connect/components/outputs/azure_data_lake_gen2/ status: certified - name: Azure Queue Storage url: /redpanda-connect/components/outputs/azure_queue_storage/ status: certified - name: Azure Table Storage url: /redpanda-connect/components/outputs/azure_table_storage/ status: certified - name: Apache Iceberg url: /redpanda-connect/components/outputs/iceberg/ status: certified - name: Social description: Outputs that write to social applications and services. items: - name: discord url: /redpanda-connect/components/outputs/discord/ status: community - name: Local description: Outputs that write to the local machine/filesystem. items: - name: file url: /redpanda-connect/components/outputs/file/ status: certified - name: stdout url: /redpanda-connect/components/outputs/stdout/ status: certified - name: Utility description: Outputs that provide utility by combining/wrapping other outputs. items: - name: broker url: /redpanda-connect/components/outputs/broker/ status: certified - name: drop url: /redpanda-connect/components/outputs/drop/ status: certified - name: drop_on url: /redpanda-connect/components/outputs/drop_on/ status: certified - name: dynamic url: /redpanda-connect/components/outputs/dynamic/ status: community - name: fallback url: /redpanda-connect/components/outputs/fallback/ status: certified - name: inproc url: /redpanda-connect/components/outputs/inproc/ status: certified - name: reject url: /redpanda-connect/components/outputs/reject/ status: certified - name: reject_errored url: /redpanda-connect/components/outputs/reject_errored/ status: certified - name: resource url: /redpanda-connect/components/outputs/resource/ status: certified - name: retry url: /redpanda-connect/components/outputs/retry/ status: certified - name: subprocess url: /redpanda-connect/components/outputs/subprocess/ status: community - name: switch url: /redpanda-connect/components/outputs/switch/ status: certified - name: sync_response url: /redpanda-connect/components/outputs/sync_response/ status: certified latest-operator-version: v26.1.2 latest-console-tag: v3.7.1 latest-redpanda-tag: v26.1.3 docname: k-connect-helm-spec page-component-name: redpanda-connect page-version: master page-component-version: master page-component-title: Connect page-relative-src-path: k-connect-helm-spec.adoc page-edit-url: https://github.com/redpanda-data/rp-connect-docs/edit/main/modules/reference/pages/k-connect-helm-spec.adoc page-git-created-date: "2024-11-14" page-git-modified-date: "2024-12-27" --- ![Version: 3.0.3](https://img.shields.io/badge/Version-3.0.3-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 4.42.0](https://img.shields.io/badge/AppVersion-4.42.0-informational?style=flat-square) Redpanda Connect is a high performance and resilient stream processor, able to connect various sources and sinks in a range of brokering patterns and perform hydration, enrichments, transformations and filters on payloads. This Helm chart deploys a Redpanda Connect instance in either streams mode or standalone. This page describes the contents of the chart’s [`values.yaml` file](https://github.com/redpanda-data/helm-charts/blob/main/charts/connect/values.yaml). Each of the settings is listed and described on this page, along with any default values. For instructions on how to install and use the chart, including how to override and customize the chart’s values, refer to the [deployment documentation](https://docs.redpanda.com/redpanda-connect/get-started/helm-chart/). > 📝 **NOTE** > > To migrate to this chart from the [Benthos Helm chart](https://github.com/redpanda-data/redpanda-connect-helm-chart), see the [migration guide in this GitHub repository](https://github.com/redpanda-data/helm-charts/blob/main/charts/connect/MIGRATION_FROM_BENTHOS.md). * * * Autogenerated from chart metadata using [helm-docs v1.14.2](https://github.com/norwoodj/helm-docs/releases/v1.14.2) ## [](#source-code)Source Code - [https://github.com/redpanda-data/helm-charts](https://github.com/redpanda-data/helm-charts) ## [](#settings)Settings ### [](#affinity)[affinity](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=affinity) Affinity rules for scheduling Pods. **Default:** `{}` ### [](#args)[args](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=args) Override the default arguments for the entrypoint command. **Default:** `[]` ### [](#autoscaling)[autoscaling](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=autoscaling) Configuration for Horizontal Pod Autoscaling for the Redpanda Connect Deployment. **Default:** {"enabled":false,"maxReplicas":12,"metrics":\[{"resource":{"name":"cpu","target":{"averageUtilization":80,"type":"Utilization"}},"type":"Resource"}\],"minReplicas":1} ### [](#autoscaling-enabled)[autoscaling.enabled](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=autoscaling.enabled) Enable Horizontal Pod Autoscaling. **Default:** `false` ### [](#autoscaling-maxreplicas)[autoscaling.maxReplicas](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=autoscaling.maxReplicas) Maximum number of replicas if autoscaling is enabled. **Default:** `12` ### [](#autoscaling-metrics)[autoscaling.metrics](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=autoscaling.metrics) List of metrics to use for scaling, such as CPU utilization. **Default:** \[{"resource":{"name":"cpu","target":{"averageUtilization":80,"type":"Utilization"}},"type":"Resource"}\] ### [](#autoscaling-minreplicas)[autoscaling.minReplicas](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=autoscaling.minReplicas) Minimum number of replicas if autoscaling is enabled. **Default:** `1` ### [](#command)[command](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=command) Override the default entrypoint command of the container. **Default:** `[]` ### [](#commonlabels)[commonLabels](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=commonLabels) Additional labels to apply to all resources created by this chart. **Default:** `{}` ### [](#config)[config](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=config) Configuration for `/redpanda-connect.yaml` file. **Default:** `{}` ### [](#deployment-annotations)[deployment.annotations](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.annotations) Set Annotations on the Redpanda Connect Deployment. **Default:** `{}` ### [](#deployment-livenessprobe)[deployment.livenessProbe](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.livenessProbe) Configuration for the liveness probe that checks if the container is running. **Default:** {"failureThreshold":3,"httpGet":{"path":"/ping","port":"http"},"periodSeconds":5,"successThreshold":1,"timeoutSeconds":2} ### [](#deployment-livenessprobe-failurethreshold)[deployment.livenessProbe.failureThreshold](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.livenessProbe.failureThreshold) Number of consecutive failures before Pod is restarted. **Default:** `3` ### [](#deployment-livenessprobe-httpget)[deployment.livenessProbe.httpGet](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.livenessProbe.httpGet) HTTP GET request configuration for the liveness probe. **Default:** {"path":"/ping","port":"http"} ### [](#deployment-livenessprobe-periodseconds)[deployment.livenessProbe.periodSeconds](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.livenessProbe.periodSeconds) Frequency (in seconds) of liveness checks. **Default:** `5` ### [](#deployment-livenessprobe-successthreshold)[deployment.livenessProbe.successThreshold](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.livenessProbe.successThreshold) Number of consecutive successes required to mark as healthy. **Default:** `1` ### [](#deployment-livenessprobe-timeoutseconds)[deployment.livenessProbe.timeoutSeconds](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.livenessProbe.timeoutSeconds) Timeout (in seconds) for each liveness check. **Default:** `2` ### [](#deployment-podannotations)[deployment.podAnnotations](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.podAnnotations) Annotations to add to Redpanda Connect Pods. **Default:** `{}` ### [](#deployment-podlabels)[deployment.podLabels](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.podLabels) Labels to add to Redpanda Connect Pods. **Default:** `{}` ### [](#deployment-readinessprobe)[deployment.readinessProbe](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.readinessProbe) Configuration for the readiness probe that checks if the container is ready to accept traffic. **Default:** {"failureThreshold":1,"httpGet":{"path":"/ready","port":"http"},"periodSeconds":5,"successThreshold":1,"timeoutSeconds":2} ### [](#deployment-readinessprobe-failurethreshold)[deployment.readinessProbe.failureThreshold](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.readinessProbe.failureThreshold) Number of consecutive failures before marking Pod as unhealthy. **Default:** `1` ### [](#deployment-readinessprobe-httpget)[deployment.readinessProbe.httpGet](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.readinessProbe.httpGet) HTTP GET request configuration for the readiness probe. **Default:** {"path":"/ready","port":"http"} ### [](#deployment-readinessprobe-periodseconds)[deployment.readinessProbe.periodSeconds](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.readinessProbe.periodSeconds) Frequency (in seconds) of readiness checks. **Default:** `5` ### [](#deployment-readinessprobe-successthreshold)[deployment.readinessProbe.successThreshold](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.readinessProbe.successThreshold) Number of consecutive successes before marking Pod as healthy. **Default:** `1` ### [](#deployment-readinessprobe-timeoutseconds)[deployment.readinessProbe.timeoutSeconds](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.readinessProbe.timeoutSeconds) Timeout (in seconds) for each readiness check. **Default:** `2` ### [](#deployment-replicacount)[deployment.replicaCount](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.replicaCount) Number of Replicas for the Redpanda Connect Deployment. **Default:** `1` ### [](#deployment-restartpolicy)[deployment.restartPolicy](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.restartPolicy) Restart policy for containers in the Pod. **Default:** `"Always"` ### [](#deployment-rolloutconfigmap)[deployment.rolloutConfigMap](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.rolloutConfigMap) Rollout the deployment on ConfigMap changes. **Default:** `true` ### [](#deployment-terminationgraceperiodseconds)[deployment.terminationGracePeriodSeconds](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=deployment.terminationGracePeriodSeconds) Time in seconds for Pod termination grace period. **Default:** `60` ### [](#env)[env](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=env) Environment variables to pass to the Redpanda Connect container. **Default:** `[]` ### [](#envfrom)[envFrom](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=envFrom) Names of Secrets or ConfigMaps that contain environment variables to pass to the Redpanda Connect container. **Default:** `[]` ### [](#extravolumemounts)[extraVolumeMounts](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=extraVolumeMounts) Additional volume mounts for the container. **Default:** `[]` ### [](#extravolumes)[extraVolumes](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=extraVolumes) Additional volumes to add to the Pod. **Default:** `[]` ### [](#fullnameoverride)[fullnameOverride](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=fullnameOverride) Override for the full name template. **Default:** `""` ### [](#http)[http](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=http) Configuration for the HTTP server used to expose Redpanda Connect endpoints. **Default:** {"address":"0.0.0.0:4195","cors":{"enabled":false},"debug\_endpoints":false,"enabled":true,"root\_path":"/redpanda-connect"} ### [](#http-address)[http.address](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=http.address) Bind Address for the HTTP server. **Default:** `"0.0.0.0:4195"` ### [](#http-cors-enabled)[http.cors.enabled](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=http.cors.enabled) Enable CORS support for HTTP server. **Default:** `false` ### [](#http-debug_endpoints)[http.debug\_endpoints](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=http.debug_endpoints) Enable debug endpoints for the HTTP server. **Default:** `false` ### [](#http-enabled)[http.enabled](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=http.enabled) Enable HTTP server. **Default:** `true` ### [](#http-root_path)[http.root\_path](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=http.root_path) Root path for HTTP API endpoints. **Default:** `"/redpanda-connect"` ### [](#image)[image](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=image) Configuration for the Redpanda Connect container image. **Default:** {"pullPolicy":"IfNotPresent","repository":"docker.redpanda.com/redpandadata/connect","tag":""} ### [](#image-pullpolicy)[image.pullPolicy](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=image.pullPolicy) Image pull policy, such as `IfNotPresent` or `Always`. **Default:** `"IfNotPresent"` ### [](#image-repository)[image.repository](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=image.repository) Docker repository for the Redpanda Connect image. **Default:** "docker.redpanda.com/redpandadata/connect" ### [](#image-tag)[image.tag](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=image.tag) Override for the default image tag (uses chart `appVersion` if empty). **Default:** `""` ### [](#imagepullsecrets)[imagePullSecrets](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=imagePullSecrets) List of Secrets for private Docker registry authentication. **Default:** `[]` ### [](#ingress-annotations)[ingress.annotations](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=ingress.annotations) Annotations for the Ingress resource. **Default:** `{}` ### [](#ingress-classname)[ingress.className](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=ingress.className) Ingress Class name for the Ingress resource. **Default:** `""` ### [](#ingress-enabled)[ingress.enabled](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=ingress.enabled) Enable Ingress resource creation. **Default:** `false` ### [](#ingress-hosts)[ingress.hosts](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=ingress.hosts) List of host definitions for the Ingress resource. **Default:** `[]` ### [](#ingress-tls)[ingress.tls](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=ingress.tls) TLS configuration for Ingress resource. **Default:** `[]` ### [](#initcontainers)[initContainers](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=initContainers) Additional init containers to add to the Deployment. **Default:** `[]` ### [](#nameoverride)[nameOverride](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=nameOverride) **Default:** `""` ### [](#nodeselector)[nodeSelector](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=nodeSelector) Node selector for scheduling Pods. **Default:** `{}` ### [](#poddisruptionbudget)[podDisruptionBudget](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=podDisruptionBudget) Configuration for the PodDisruptionBudget, which controls voluntary evictions of Redpanda Connect Pods. **Default:** `{"enabled":false}` ### [](#poddisruptionbudget-enabled)[podDisruptionBudget.enabled](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=podDisruptionBudget.enabled) Enable a PodDisruptionBudget for the Deployment. **Default:** `false` ### [](#podsecuritycontext)[podSecurityContext](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=podSecurityContext) Security context to be applied at the Pod level. **Default:** `{}` ### [](#resources)[resources](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=resources) Resource limits and requests for the container. **Default:** `{}` ### [](#securitycontext)[securityContext](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=securityContext) Security context to be applied to containers in the Pod. **Default:** `{}` ### [](#service)[service](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=service) Configuration for the Kubernetes Service that exposes Redpanda Connect. **Default:** {"extraPorts":null,"name":"http","port":80,"protocol":"TCP","targetPort":"http","type":"ClusterIP"} ### [](#service-extraports)[service.extraPorts](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=service.extraPorts) Additional ports to expose through the Service. **Default:** `nil` ### [](#service-name)[service.name](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=service.name) Name of the Kubernetes Service. **Default:** `"http"` ### [](#service-port)[service.port](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=service.port) Service port for accessing the application. **Default:** `80` ### [](#service-protocol)[service.protocol](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=service.protocol) Protocol to use for the Service, for example TCP or UDP. **Default:** `"TCP"` ### [](#service-targetport)[service.targetPort](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=service.targetPort) Target port on the container. **Default:** `"http"` ### [](#service-type)[service.type](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=service.type) Type of Kubernetes Service, for example ClusterIP, NodePort, or LoadBalancer. **Default:** `"ClusterIP"` ### [](#serviceaccount)[serviceAccount](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=serviceAccount) Configuration for the Kubernetes ServiceAccount associated with the Redpanda Connect Pods. **Default:** {"annotations":{},"create":true,"name":""} ### [](#serviceaccount-annotations)[serviceAccount.annotations](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=serviceAccount.annotations) Annotations to add to the ServiceAccount. **Default:** `{}` ### [](#serviceaccount-create)[serviceAccount.create](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=serviceAccount.create) Specify whether a ServiceAccount should be created. **Default:** `true` ### [](#serviceaccount-name)[serviceAccount.name](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=serviceAccount.name) Name of the ServiceAccount to use. If not set and `create` is `true`, a name is autogenerated. **Default:** `""` ### [](#servicemonitor)[serviceMonitor](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=serviceMonitor) Configuration for the ServiceMonitor used by Prometheus to scrape Redpanda Connect metrics. **Default:** {"enabled":false,"interval":"10s","scheme":"http"} ### [](#servicemonitor-enabled)[serviceMonitor.enabled](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=serviceMonitor.enabled) Enable the creation of a ServiceMonitor for Prometheus monitoring. **Default:** `false` ### [](#servicemonitor-interval)[serviceMonitor.interval](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=serviceMonitor.interval) Scrape interval for Prometheus monitoring. **Default:** `"10s"` ### [](#servicemonitor-scheme)[serviceMonitor.scheme](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=serviceMonitor.scheme) HTTP Scheme to use for scraping. **Default:** `"http"` ### [](#streams)[streams](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=streams) Configuration settings for the streams mode feature in Redpanda Connect. **Default:** {"api":{"enable":true},"enabled":false,"streamsConfigMap":""} ### [](#streams-api-enable)[streams.api.enable](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=streams.api.enable) Enable the streams API. **Default:** `true` ### [](#streams-enabled)[streams.enabled](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=streams.enabled) Enable streams mode. **Default:** `false` ### [](#streams-streamsconfigmap)[streams.streamsConfigMap](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=streams.streamsConfigMap) Name of the ConfigMap that contains stream configuration files. **Default:** `""` ### [](#telemetry)[telemetry](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=telemetry) **Default:** `true` ### [](#tolerations)[tolerations](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=tolerations) Tolerations for scheduling Pods onto nodes with matching taints. **Default:** `[]` ### [](#topologyspreadconstraints)[topologySpreadConstraints](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=topologySpreadConstraints) **Default:** `[]` ### [](#updatestrategy)[updateStrategy](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=updateStrategy) Update strategy for the Deployment, for example `RollingUpdate`. **Default:** `{}` ### [](#watch)[watch](https://artifacthub.io/packages/helm/redpanda-data/connect?modal=values&path=watch) Experimental feature: Watch config files for changes and automatically apply them without needing to rollout the Deployment. Make sure that `deployment.rolloutConfigMap` is `false`. **Default:** `false` ---